Skip to content

Commit 2ce3ba5

Browse files
jacobecoxKyle CuppJakob Nagelhakan-controlplane
authored
added cassandra template for single location with backups and repair (#259)
* init 1.0.0 * replicas successfully connecting * init script now queries for sentinel for master, updated components around that feature, added cpln-common tags (#244) * init 3.3.0 * added sentinel password env to redis workload, init now queries sentinel for master * added cpln-common tagging * added sentinel secret to redis policy * added master discovery on publicAccess mode * working changes * fixed probes * added tagging names, cpln-common tags * added post stop hook to handle shut down * added multi location support * improved ha proxy health check and patroni resiliancy with dcs (#245) * init 2.3.1 * switched ha proxy to use http over tcp check, added 2 health endpoints for ha proxy, increased dcs retry limit * lowered proxy rise to 1 (#246) * added secondary index value, added publishNotReadyAddresses tag, moved searcbd log dir (#247) * init 2.0.1 * added secondary index value * added secondary indexes * moved searchd log directory to prevent loading wrong config from donor * added publish not ready addresses tag * defaulted secondary indexes to false * lowered proxy rise to 1, added liveness probe, readiness probe and prestop hook (#248) * lowered proxy rise to 1 * added prestop hook, liveness probe and readiness probe * added secondary index value, added publishNotReadyAddresses tag, moved searcbd log dir (#247) * init 2.0.1 * added secondary index value * added secondary indexes * moved searchd log directory to prevent loading wrong config from donor * added publish not ready addresses tag * defaulted secondary indexes to false * readded to template (#249) * readded to template * added cpln-common tags * redis replica init now queries sentinel for master on startup (#250) * init 3.4.0 * public access now uses internal replica routing instead of external through domain * sentinel properly announces right hostname * extended sentinel timeouts, switched sentinel announce-ip to use headless * switched announce ip on sentinel to replica direct naming, changed domains to only use single port * improved ha proxy health check and patroni resiliancy with dcs (#245) * init 2.3.1 * switched ha proxy to use http over tcp check, added 2 health endpoints for ha proxy, increased dcs retry limit * lowered proxy rise to 1 (#246) * added secondary index value, added publishNotReadyAddresses tag, moved searcbd log dir (#247) * init 2.0.1 * added secondary index value * added secondary indexes * moved searchd log directory to prevent loading wrong config from donor * added publish not ready addresses tag * defaulted secondary indexes to false * lowered proxy rise to 1, added liveness probe, readiness probe and prestop hook (#248) * lowered proxy rise to 1 * added prestop hook, liveness probe and readiness probe * added secondary index value, added publishNotReadyAddresses tag, moved searcbd log dir (#247) * init 2.0.1 * added secondary index value * added secondary indexes * moved searchd log directory to prevent loading wrong config from donor * added publish not ready addresses tag * defaulted secondary indexes to false * readded to template (#249) * readded to template * added cpln-common tags * updated chart file * added retry for redis getting master from sentinel, updated non publicaccess mode to query sentinel hostname not replica specific sentinel * updated readme and releases files (#251) * updated public access sentinel querying logic (#252) * added 1password connect provider (#253) * init 1.4.0 * added 1password connect provider * added cpln-common tagging * added 1password connect (#254) * redis v3.2.0 - add knobs for repl-backlog-size and repl-timeout. Important for clusters with very high throughput and/or large data sets - add knob for client-output-buffer-limit to avoid problems during long full resyncs. - add cpln/publishNotReadyAddresses: "true" for both redis and sentinel so bootstrapping is always possible even when replicas aren't yet ready - add a more sophisticated readiness probe such that replicas are only ready once they're caught up to the master. - add a knob for requestRetryPolicy * updated template config * updated values file * updated startup script * updated script so all replicas are seeds * reverted to single location * updated node communication port, reverted to multi location setup * updated listen address and broadcast address * debezium + cdc pipeline templates * redis: apply replication setting tweaks from 3.2.0 to 3.3.0. * redis v3.2.0: scalingPolicy: Parallel * ESS version 1.3.5 * cdc-pipeline: add icon * redis: liveness probe must be a TCP port check only * postgres-ha: multi-dc support. * kafka: v4.0.0 * redis updates - improve resilience in v3.2.0 - v3.3.0 is not v3.2.0 + master discovery on startup for redis replicas. * cdc-pipeline: use kafka v4.0.0 * kafka: fix pipeline * ESS: gcp sync all secrets * shorten cdc desc * added pgbouncer and probes, fixed init bugs, added single location support, updated readme (#255) * init 1.4.0 * init script now runs in foreground and runs exec * removed extra wait * added liveness and readiness probes * added prestop hook and removed probes temporarily * readded probes * added pgbouncer * added cpln-common, removed tag * added probes, added pgbouncer, fixed startup bugs, supports single location, added cpln-common * updated README * added firewall for pgbouncer settings * added cockroach to its firewall list when pgbouncer is enabled (#256) * updated headers patch to not touch other settings (#257) * init 1.1.1 * added cockroach to its firewall list when pgbouncer is enabled (#256) * updated patch script * updated patch to be surgical on headers setting * reverted to single location * updated hostname in script * added auth + database credentials and replication factor * moved wait for repair system_auth * reset default values * moved validation to helpers file, added repair cron job * updated cron job * added multizone * updated backup image and functionality with pgbouncer, added multizone support (#258) * updated backup image, added logic to point to pgbouncer * added multizone for cockroach * updated backup image, added multizone per local option * init backup * logical backup working state * physical backup added * opened firewall for backup and restore on cassandra workload * added readme * updated readme * changed createGvc to false --------- Co-authored-by: Kyle Cupp <kyle@controlplane.com> Co-authored-by: Jakob Nagel <jakob@controlplane.com> Co-authored-by: Hakan <hakan@controlplane.com>
1 parent 3c18cd0 commit 2ce3ba5

14 files changed

Lines changed: 904 additions & 0 deletions

cassandra/icon.png

185 KB
Loading
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: v2
2+
name: cassandra
3+
description: Cassandra cluster for Control Plane
4+
type: application
5+
version: 1.0.0
6+
appVersion: "5.0"
7+
8+
annotations:
9+
created: "2026-05-18"
10+
lastModified: "2026-05-18"
11+
category: "database"
12+
createsGvc: false
13+
14+
dependencies:
15+
- name: cpln-common
16+
version: 1.0.0
17+
repository: "oci://ghcr.io/controlplane-com/templates"

cassandra/versions/1.0.0/README.md

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
# Cassandra
2+
3+
This app deploys a Cassandra 5.0 cluster in a single location. Each node runs as a stateful replica with its own persistent volume, forming a peer-to-peer cluster that distributes and replicates data across nodes according to the configured replication factor. The template includes optional scheduled backups (logical or physical) and periodic anti-entropy repair.
4+
5+
## Architecture
6+
7+
- **Cassandra cluster**: Multi-node cluster deployed in a single location where each node owns a slice of the token ring and replicates data to peers
8+
- **Per-node volumes**: Each node gets its own persistent volume so SSTable data survives restarts
9+
- **Repair** (optional): Scheduled cron job that runs `nodetool repair` across all nodes to keep data consistent
10+
- **Backup** (optional): Logical (`cqlsh COPY TO`) or physical (`nodetool snapshot`) backup to S3 or GCS
11+
12+
## Configuration
13+
14+
### Core Settings
15+
16+
```yaml
17+
replicas: 3 # Number of Cassandra nodes
18+
replicationFactor: 3 # Copies of each partition stored across the cluster
19+
# Must not exceed replicas
20+
21+
superuserPassword: supersecretpassword # Built-in cassandra superuser password
22+
username: username # Application user
23+
password: password # Application user password
24+
keyspaceName: mydatabase # Keyspace created on startup
25+
26+
image: cassandra:5.0
27+
cpu: 1
28+
memory: 4Gi
29+
jvmHeapSize: 2G # Set to ~50% of memory — Cassandra needs the rest for off-heap cache
30+
clusterName: my-cassandra
31+
```
32+
33+
**Volume** — set the initial storage capacity and optionally enable autoscaling:
34+
35+
```yaml
36+
volumes:
37+
data:
38+
initialCapacity: 10 # GiB
39+
autoscaling:
40+
maxCapacity: 100
41+
minFreePercentage: 20
42+
scalingFactor: 1.5
43+
```
44+
45+
Configure which workloads can reach Cassandra:
46+
47+
```yaml
48+
internal_access:
49+
type: same-gvc # Options: same-gvc, same-org, workload-list
50+
workloads:
51+
# Uncomment and specify workloads if using workload-list
52+
#- //gvc/GVC_NAME/workload/WORKLOAD_NAME
53+
```
54+
55+
- `same-gvc`: Allow access from all workloads in the same GVC
56+
- `same-org`: Allow access from all workloads in the org
57+
- `workload-list`: Allow access only from specified workloads
58+
59+
## Connecting
60+
61+
Each Cassandra replica is reachable via its own DNS name:
62+
63+
```
64+
Host: {release-name}-cassandra-{n}.{gvc}.cpln.local
65+
Port: 9042 (CQL, native transport)
66+
Username: {username}
67+
Password: {password}
68+
Keyspace: {keyspaceName}
69+
```
70+
71+
Provide multiple node hostnames as contact points in your application so it can discover the full cluster topology.
72+
73+
## Replicas vs Replication Factor
74+
75+
These are two separate settings that work together:
76+
77+
- **`replicas`** — how many Cassandra nodes are deployed. More nodes means more capacity and better throughput, as the token ring is split across more nodes.
78+
- **`replicationFactor`** — how many copies of each partition are stored across the cluster. A replication factor of 3 means every row exists on 3 different nodes, so the cluster can survive 2 node failures without data loss (with `QUORUM` consistency).
79+
80+
`replicationFactor` must not exceed `replicas` — you cannot store 3 copies of data across only 2 nodes.
81+
82+
## Multi-Zone
83+
84+
When `multiZone.enabled: true`, Control Plane spreads replicas across availability zones within the location:
85+
86+
```yaml
87+
multiZone:
88+
enabled: true
89+
```
90+
91+
With a replication factor of 3 across 3 zones, each zone holds one copy of every partition. The cluster survives a complete zone outage with no data loss, provided your client uses `LOCAL_QUORUM` consistency (reads and writes succeed with responses from the surviving 2 zones).
92+
93+
Verify your selected location supports multi-zone before enabling this option.
94+
95+
## Repair
96+
97+
Cassandra uses eventual consistency — when nodes miss writes during downtime, data can drift out of sync. `nodetool repair` runs an anti-entropy process that compares and reconciles data across all replicas. Repair must complete across all nodes at least once within `gc_grace_seconds` (default: 10 days) to prevent deleted data from reappearing.
98+
99+
The template includes a scheduled repair cron job:
100+
101+
```yaml
102+
repair:
103+
enabled: true
104+
schedule: "0 2 * * 0" # Weekly, Sunday at 2am UTC
105+
```
106+
107+
The default weekly schedule satisfies the 10-day `gc_grace_seconds` requirement with margin. Do not disable repair in production or increase the interval beyond 10 days.
108+
109+
Repair can be resource-intensive on large datasets. If it impacts query performance, consider running it during low-traffic windows or increasing node resources.
110+
111+
## Backing Up
112+
113+
Two backup modes are available:
114+
115+
- **Logical** — exports keyspace tables as CSVs using `cqlsh COPY TO`, then uploads to cloud storage. Runs as a standalone cron workload on schedule. Suitable for smaller datasets or when portability matters.
116+
- **Physical** — creates SSTable snapshots using `nodetool snapshot` and syncs them to cloud storage. Runs as a sidecar container on each Cassandra replica. Faster and more space-efficient for large datasets, but backups are per-node and must be restored node-by-node.
117+
118+
Set `backup.enabled: true`, choose a `type`, set `backup.provider`, and fill in the corresponding cloud block:
119+
120+
```yaml
121+
backup:
122+
enabled: true
123+
type: logical # logical or physical
124+
image: ghcr.io/controlplane-com/backup-images/cassandra-backup:5.0
125+
schedule: "0 2 * * *" # daily at 2am UTC
126+
127+
resources:
128+
cpu: 250m
129+
memory: 256Mi
130+
131+
provider: aws # aws or gcp
132+
133+
aws:
134+
bucket: my-backup-bucket
135+
region: us-east-1
136+
cloudAccountName: my-backup-cloudaccount
137+
policyName: my-s3-policy
138+
prefix: cassandra/backups
139+
140+
gcp:
141+
bucket: my-backup-bucket
142+
cloudAccountName: my-cloud-account
143+
prefix: cassandra/backups
144+
```
145+
146+
### AWS S3
147+
148+
1. Create your S3 bucket. Set `aws.bucket` and `aws.region` to match.
149+
150+
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](https://docs.controlplane.com/guides/create-cloud-account). Set `aws.cloudAccountName` to match.
151+
152+
3. Create an AWS IAM policy with the following JSON (replace `YOUR_BUCKET_NAME`):
153+
154+
```json
155+
{
156+
"Version": "2012-10-17",
157+
"Statement": [
158+
{
159+
"Effect": "Allow",
160+
"Action": [
161+
"s3:GetObject",
162+
"s3:PutObject",
163+
"s3:DeleteObject",
164+
"s3:ListBucket",
165+
"s3:GetObjectVersion",
166+
"s3:DeleteObjectVersion"
167+
],
168+
"Resource": [
169+
"arn:aws:s3:::YOUR_BUCKET_NAME",
170+
"arn:aws:s3:::YOUR_BUCKET_NAME/*"
171+
]
172+
}
173+
]
174+
}
175+
```
176+
177+
4. Set `aws.policyName` to the name of the policy created in step 3.
178+
179+
### GCS
180+
181+
1. Create your GCS bucket. Set `gcp.bucket` to match.
182+
183+
2. If you do not have a Cloud Account set up, refer to the docs to [Create a Cloud Account](https://docs.controlplane.com/guides/create-cloud-account). Set `gcp.cloudAccountName` to match.
184+
185+
**Important**: Add the `Storage Admin` role to the GCP service account created for the Cloud Account.
186+
187+
## Restoring a Backup
188+
189+
### Logical Restore
190+
191+
Exec into the backup cron workload and run `restore.sh` with the timestamp of the backup you want to restore:
192+
193+
```bash
194+
RESTORE_TIMESTAMP=2026-05-15T02-00-00Z /usr/local/bin/restore.sh
195+
```
196+
197+
The timestamp format matches the backup filename in your bucket (e.g. `cassandra/backups/2026-05-15T02-00-00Z/`).
198+
199+
The script downloads the CSVs for the configured keyspace and replays them into Cassandra using `cqlsh COPY FROM`. Existing rows with matching primary keys are overwritten; rows not in the backup are left in place.
200+
201+
### Physical Restore
202+
203+
Physical backups are per-node — each replica backed up its own SSTable slice. To restore, exec into the **backup sidecar container** (not the cassandra container) on each replica that needs to be restored and run:
204+
205+
```bash
206+
RESTORE_TIMESTAMP=2026-05-15T02-00-00Z /usr/local/bin/restore.sh
207+
```
208+
209+
The script downloads the snapshot files for that replica from `{prefix}/{timestamp}/{hostname}/`, writes them to the shared volume, then calls `nodetool import` to load the SSTables into the live Cassandra instance without a restart.
210+
211+
**Important**: Repeat this on every replica. Because each node owns a different token range, restoring only one replica leaves the cluster with incomplete data.
212+
213+
## Important Notes
214+
215+
- **Minimum replicas for production**: Use at least 3 replicas with a replication factor of 3 so the cluster can survive a node failure while still achieving quorum
216+
- **JVM heap**: Set `jvmHeapSize` to approximately 50% of `memory` — Cassandra relies heavily on off-heap memory for bloom filters, row cache, and OS page cache
217+
- **gc_grace_seconds**: The default is 10 days. Ensure repair runs at least once within this window on all nodes, or deleted data may reappear after a node recovers from downtime
218+
- **Scaling up**: Adding replicas after initial deployment does not automatically rebalance data. Run `nodetool rebuild` on new nodes and then `nodetool cleanup` on existing nodes after scaling
219+
- **Multi-zone**: Verify your selected location supports multi-zone before enabling
220+
221+
## Supported External Services
222+
223+
- [Cassandra Documentation](https://cassandra.apache.org/doc/latest/)
224+
- [Cassandra Driver Documentation](https://docs.datastax.com/en/developer/driver-matrix/doc/common/driverMatrix.html)
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
{{/* Resource Naming */}}
2+
3+
{{- define "cassandra.workload.name" -}}
4+
{{- printf "%s-cassandra" .Release.Name }}
5+
{{- end }}
6+
7+
{{- define "cassandra.secret.init.name" -}}
8+
{{- printf "%s-cassandra-init" .Release.Name }}
9+
{{- end }}
10+
11+
{{- define "cassandra.secret.config.name" -}}
12+
{{- printf "%s-cassandra-config" .Release.Name }}
13+
{{- end }}
14+
15+
{{- define "cassandra.identity.name" -}}
16+
{{- printf "%s-cassandra-identity" .Release.Name }}
17+
{{- end }}
18+
19+
{{- define "cassandra.policy.name" -}}
20+
{{- printf "%s-cassandra-policy" .Release.Name }}
21+
{{- end }}
22+
23+
{{- define "cassandra.volumeset.name" -}}
24+
{{- printf "%s-cassandra-data" .Release.Name }}
25+
{{- end }}
26+
27+
{{- define "cassandra.secret.credentials.name" -}}
28+
{{- printf "%s-cassandra-credentials" .Release.Name }}
29+
{{- end }}
30+
31+
{{- define "cassandra.workload.repair.name" -}}
32+
{{- printf "%s-cassandra-repair" .Release.Name }}
33+
{{- end }}
34+
35+
{{- define "cassandra.workload.backup.name" -}}
36+
{{- printf "%s-cassandra-backup" .Release.Name }}
37+
{{- end }}
38+
39+
40+
{{/* Validation */}}
41+
42+
{{- define "cassandra.validate" -}}
43+
{{- if gt (.Values.replicationFactor | int) (.Values.replicas | int) }}
44+
{{- fail (printf "replicationFactor (%d) cannot exceed replicas (%d)" (.Values.replicationFactor | int) (.Values.replicas | int)) }}
45+
{{- end }}
46+
{{- if .Values.backup.enabled }}
47+
{{- if not (or (eq .Values.backup.type "logical") (eq .Values.backup.type "physical")) }}
48+
{{- fail (printf "backup.type must be 'logical' or 'physical', got: %s" .Values.backup.type) }}
49+
{{- end }}
50+
{{- if not (or (eq .Values.backup.provider "aws") (eq .Values.backup.provider "gcp")) }}
51+
{{- fail (printf "backup.provider must be 'aws' or 'gcp', got: %s" .Values.backup.provider) }}
52+
{{- end }}
53+
{{- if eq .Values.backup.provider "aws" }}
54+
{{- if not .Values.backup.aws.cloudAccountName }}
55+
{{- fail "backup.aws.cloudAccountName is required when backup.provider is aws" }}
56+
{{- end }}
57+
{{- if not .Values.backup.aws.policyName }}
58+
{{- fail "backup.aws.policyName is required when backup.provider is aws" }}
59+
{{- end }}
60+
{{- if not .Values.backup.aws.bucket }}
61+
{{- fail "backup.aws.bucket is required when backup.provider is aws" }}
62+
{{- end }}
63+
{{- end }}
64+
{{- if eq .Values.backup.provider "gcp" }}
65+
{{- if not .Values.backup.gcp.cloudAccountName }}
66+
{{- fail "backup.gcp.cloudAccountName is required when backup.provider is gcp" }}
67+
{{- end }}
68+
{{- if not .Values.backup.gcp.bucket }}
69+
{{- fail "backup.gcp.bucket is required when backup.provider is gcp" }}
70+
{{- end }}
71+
{{- end }}
72+
{{- end }}
73+
{{- end }}
74+
75+
76+
{{/* Labeling */}}
77+
78+
{{- define "cassandra.tags" -}}
79+
{{- include "cpln-common.tags" . }}
80+
{{- end }}
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
kind: identity
2+
name: {{ include "cassandra.identity.name" . }}
3+
description: {{ include "cassandra.workload.name" . }} identity
4+
tags: {{- include "cassandra.tags" . | nindent 2 }}
5+
{{- if and .Values.backup.enabled (eq .Values.backup.provider "aws") }}
6+
aws:
7+
cloudAccountLink: //cloudaccount/{{ .Values.backup.aws.cloudAccountName }}
8+
policyRefs:
9+
- cpln-connector
10+
- aws::ReadOnlyAccess
11+
- {{ .Values.backup.aws.policyName | quote }}
12+
{{- end }}
13+
{{- if and .Values.backup.enabled (eq .Values.backup.provider "gcp") }}
14+
gcp:
15+
bindings:
16+
- resource: //storage.googleapis.com/projects/_/buckets/{{ .Values.backup.gcp.bucket }}
17+
roles:
18+
- roles/storage.objectAdmin
19+
cloudAccountLink: //cloudaccount/{{ .Values.backup.gcp.cloudAccountName }}
20+
scopes:
21+
- https://www.googleapis.com/auth/cloud-platform
22+
{{- end }}
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
kind: policy
2+
name: {{ include "cassandra.policy.name" . }}
3+
origin: default
4+
bindings:
5+
- permissions:
6+
- reveal
7+
principalLinks:
8+
- //gvc/{{ .Values.global.cpln.gvc }}/identity/{{ include "cassandra.identity.name" . }}
9+
targetKind: secret
10+
targetLinks:
11+
- //secret/{{ include "cassandra.secret.init.name" . }}
12+
- //secret/{{ include "cassandra.secret.config.name" . }}
13+
- //secret/{{ include "cassandra.secret.credentials.name" . }}

0 commit comments

Comments
 (0)