Repository navigation
Sharded clusters: spec.shards, a StatefulSet per shard, ClusterIP Services, config in a Secret - #11
Closed
murtazayusuf wants to merge 1 commit into
Closed
murtazayusuf wants to merge 1 commit into
murtazayusuf wants to merge 1 commit into
Conversation
…vices, config in a Secret - spec.shards (fixed at creation by a CEL rule; the count can change) and spec.cluster; status.shards, status.cluster, shard and node ID per pod; the Rebalancing phase. Deepcopy and CRDs edited by hand. - One StatefulSet per shard (<cluster>-<shard>) with --cluster-announce-ip $(POD_IP), preferred anti-affinity per shard, and a PodDisruptionBudget per shard. - ClusterIP Services only: per pod, per shard, replicas, the whole cluster, and a headless Service per shard (the only one with the agent port). Services of removed pods are cleaned up; user Services are not. - The rendered kividb.conf is a Secret for every cluster; in cluster mode it carries the cluster-mode directives, the bus secret (generated when unset) and masterauth. The old ConfigMap is removed once no pod mounts it. - Validation: 24-node cap, 40-character names, operator-owned cluster directives refused in a KividbConfig; DbOps restarts refused for sharded clusters for now. Forming the cluster comes next.
This was referenced Oct 24, 2026
| st.Role = kividbv1alpha1.RoleUnknown | ||
| } | ||
| if s, err := strconv.Atoi(p.Labels[ShardLabel]); err == nil { | ||
| shard := int32(s) |
This was referenced Oct 24, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10: review that first; this PR's diff is against it.
Second step of 0.7.0: the shape of a sharded cluster. A
KividbClusterwithspec.shardsgets its shards' workloads, Services, disruption budgets and cluster-mode config. Forming the cluster (CLUSTER MEET, slots, replicas), failover reporting, rollouts, backups and resizing come in the next PRs. Until then a sharded cluster staysProvisioning, with its pods running standalone in cluster mode andReadyreadingClusterNotFormed. A replication cluster is unchanged, except that its config moves into a Secret (below).API
spec.shards(≥ 1): unset means replication mode, as before. A CEL rule,has(self.shards) == has(oldSelf.shards), keeps a cluster sharded or not for good. The count itself can change.spec.cluster(only withshards):nodeTimeoutMs(default 5000),requireFullCoverage(default true),minReplicasToWrite,minReplicasMaxLagSeconds(default 10),busSecretRef(generated into<cluster>-buswhen unset).spec.replicasnow means replicas per master: beside the single master, or in each shard.status.shards[](index, master pod and node ID, slots, pods, ready pods),status.cluster(CLUSTER INFO), andshardandnodeIDon each pod. The phase enum gainsRebalancing, and there's a newShardsprinter column.Validation, in the controller too, because Helm never upgrades CRDs:
shards × (replicas + 1)), and a name of at most 40 characters;bootstrapFromSnapshotyet, andspec.clusteronly withshards;KividbConfigmay not set the directives the operator owns:cluster-*,min-replicas-*andmin-slaves-*,replicaof,masterauth/masteruser,tls-cluster.Workloads
<cluster>-<shard>, with pods<cluster>-<shard>-<n>:replicas + 1pods each, withkividb.io/shardin the selector;OnDelete, as today;--cluster-announce-ip $(POD_IP). Peers reach a node at its pod IP, and the setting is fixed at startup.spec.affinitystill replaces it entirely.maxUnavailable: 1. New RBAC:policy/poddisruptionbudgets, in the chart,config/rbacand the kubebuilder marker.KividbDbOpsrestarts of a sharded cluster are refused until the rollout step.Services: ClusterIP only
For every Service of a sharded cluster,
kividb.io/servicesays which kind it is:<cluster><cluster>-shard-<i>i<cluster>-replicas<pod>(one per pod)statefulset.kubernetes.io/pod-name<cluster>-<i>-headlessi, not-ready included: each pod's DNS name, and the only Service with the agent portConfig in a Secret, for every cluster
kividb.confis now a Secret (<cluster>-config), not a ConfigMap, because a sharded cluster's carriescluster-bus-secretandmasterauth.cluster-enabled yes,cluster-config-file nodes.conf(on the data volume) andcluster-node-timeout;cluster-require-full-coverageandcluster-preferred-endpoint-type hostname(the agent will set the announced hostname);min-replicas-*;cluster-bus-secret;masteruser defaultandmasterauth, since replicas connect on their own in cluster mode, after a failover too.reconcileStatefulSetbecameapplyStatefulSet, shared by both modes.Tested
go build,go vet,go test ./...,go mod tidy(no diff),helm lint,helm template.sharded_test.go):bootstrapFromSnapshot,spec.clusterwithout shards, the operator-owned directives;masterauthwithout a password and no cluster lines in replication mode;POD_IP, the config Secret volume, anti-affinity;