Skip to content

Sharded clusters: spec.shards, a StatefulSet per shard, ClusterIP Services, config in a Secret - #11

Closed
murtazayusuf wants to merge 1 commit into
feat/0.7.0-agent-tokenfrom
feat/0.7.0-shards-api
Closed

murtazayusuf wants to merge 1 commit into
feat/0.7.0-agent-tokenfrom
feat/0.7.0-shards-api

Conversation

@murtazayusuf

Copy link
Copy Markdown
Contributor

Stacked on #10: review that first; this PR's diff is against it.

Second step of 0.7.0: the shape of a sharded cluster. A KividbCluster with spec.shards gets its shards' workloads, Services, disruption budgets and cluster-mode config. Forming the cluster (CLUSTER MEET, slots, replicas), failover reporting, rollouts, backups and resizing come in the next PRs. Until then a sharded cluster stays Provisioning, with its pods running standalone in cluster mode and Ready reading ClusterNotFormed. A replication cluster is unchanged, except that its config moves into a Secret (below).

API

  • spec.shards (≥ 1): unset means replication mode, as before. A CEL rule, has(self.shards) == has(oldSelf.shards), keeps a cluster sharded or not for good. The count itself can change.
  • spec.cluster (only with shards): nodeTimeoutMs (default 5000), requireFullCoverage (default true), minReplicasToWrite, minReplicasMaxLagSeconds (default 10), busSecretRef (generated into <cluster>-bus when unset).
  • spec.replicas now means replicas per master: beside the single master, or in each shard.
  • Status: status.shards[] (index, master pod and node ID, slots, pods, ready pods), status.cluster (CLUSTER INFO), and shard and nodeID on each pod. The phase enum gains Rebalancing, and there's a new Shards printer column.
  • Generated code and CRDs: the deepcopy code and both CRD copies are hand-edited, as the repo does; there's no controller-gen run.

Validation, in the controller too, because Helm never upgrades CRDs:

  • at most 24 nodes (shards × (replicas + 1)), and a name of at most 40 characters;
  • no bootstrapFromSnapshot yet, and spec.cluster only with shards;
  • in a sharded cluster, a KividbConfig may not set the directives the operator owns: cluster-*, min-replicas-* and min-slaves-*, replicaof, masterauth/masteruser, tls-cluster.

Workloads

  • One StatefulSet per shard, <cluster>-<shard>, with pods <cluster>-<shard>-<n>:
    • replicas + 1 pods each, with kividb.io/shard in the selector;
    • OnDelete, as today;
    • the same pod template as a replication cluster, plus --cluster-announce-ip $(POD_IP). Peers reach a node at its pod IP, and the setting is fixed at startup.
  • Anti-affinity is preferred, not required: weight 100 against the shard's own pods, 50 against the cluster's. The design said required within a shard, but that would stop single-node and small dev clusters from scheduling. spec.affinity still replaces it entirely.
  • A PodDisruptionBudget per shard with maxUnavailable: 1. New RBAC: policy/poddisruptionbudgets, in the chart, config/rbac and the kubebuilder marker.
  • KividbDbOps restarts of a sharded cluster are refused until the rollout step.

Services: ClusterIP only

For every Service of a sharded cluster, kividb.io/service says which kind it is:

Service Selects
<cluster> every pod: a seed address for cluster clients
<cluster>-shard-<i> shard i
<cluster>-replicas every replica
<pod> (one per pod) statefulset.kubernetes.io/pod-name
<cluster>-<i>-headless shard i, not-ready included: each pod's DNS name, and the only Service with the agent port
  • No load balancers: the operator never creates one. Users expose a pod with a Service of their own and the same selector.
  • Cleanup: when replicas or shards are removed, the operator deletes its own Services that are no longer wanted. It never touches Services it doesn't control.

Config in a Secret, for every cluster

  • Why: the rendered kividb.conf is now a Secret (<cluster>-config), not a ConfigMap, because a sharded cluster's carries cluster-bus-secret and masterauth.
  • In cluster mode it adds:
    • cluster-enabled yes, cluster-config-file nodes.conf (on the data volume) and cluster-node-timeout;
    • cluster-require-full-coverage and cluster-preferred-endpoint-type hostname (the agent will set the announced hostname);
    • min-replicas-*;
    • cluster-bus-secret;
    • masteruser default and masterauth, since replicas connect on their own in cluster mode, after a failover too.
  • Upgrade:
    • Existing clusters' pods move to the Secret as they roll.
    • The old ConfigMap is deleted only once no pod of the cluster mounts it, since a pod of the old template couldn't start without it.
    • reconcileStatefulSet became applyStatefulSet, shared by both modes.

Tested

  • go build, go vet, go test ./..., go mod tidy (no diff), helm lint, helm template.
  • New tests (sharded_test.go):
    • validation: node cap, name length, bootstrapFromSnapshot, spec.cluster without shards, the operator-owned directives;
    • the cluster-mode config, including no masterauth without a password and no cluster lines in replication mode;
    • the shard StatefulSet: name, size, headless name, shard selector, the announce-IP flag and POD_IP, the config Secret volume, anti-affinity;
    • every Service: names, ClusterIP, the agent port only on the headless ones, the per-pod and shard selectors, the PDB;
    • a reconcile against a fake API server: 3 StatefulSets, 3 PDBs and 14 Services; the bus secret generated once and kept; the config carrying it; status; and, after scaling replicas down, the removed pods' Services deleted while a user's LoadBalancer with a pod selector is left alone;
    • the legacy ConfigMap kept while a pod mounts it and removed after, with the config Secret in place.
  • Not yet: an end-to-end run. The sharding e2e suite comes before the release.

…vices, config in a Secret

- spec.shards (fixed at creation by a CEL rule; the count can change)
  and spec.cluster; status.shards, status.cluster, shard and node ID per
  pod; the Rebalancing phase. Deepcopy and CRDs edited by hand.
- One StatefulSet per shard (<cluster>-<shard>) with
  --cluster-announce-ip $(POD_IP), preferred anti-affinity per shard,
  and a PodDisruptionBudget per shard.
- ClusterIP Services only: per pod, per shard, replicas, the whole
  cluster, and a headless Service per shard (the only one with the agent
  port). Services of removed pods are cleaned up; user Services are not.
- The rendered kividb.conf is a Secret for every cluster; in cluster mode
  it carries the cluster-mode directives, the bus secret (generated when
  unset) and masterauth. The old ConfigMap is removed once no pod mounts
  it.
- Validation: 24-node cap, 40-character names, operator-owned cluster
  directives refused in a KividbConfig; DbOps restarts refused for
  sharded clusters for now.

Forming the cluster comes next.
st.Role = kividbv1alpha1.RoleUnknown
}
if s, err := strconv.Atoi(p.Labels[ShardLabel]); err == nil {
shard := int32(s)
@murtazayusuf

Copy link
Copy Markdown
Contributor Author

Merged into master as part of #19 (merge commit 4263894), which carried the whole 0.7.0 stack. Closing as included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants