[murtazayusuf]

Stacked on #13 (then #12, #11, #10). The diff is against #13.

This is the fifth step of 0.7.0: sharded clusters get rolling restarts. Until now a shard's StatefulSet (OnDelete) never replaced its pods, so a template change, a new password or a config change didn't take effect, and KividbDbOps refused sharded clusters.

Rollout (shard_rollout.go)

The rules are the same as for a replicated set (reconcileRollout), applied across the whole cluster:

• One pod at a time, cluster-wide.
• Only while the cluster is whole: formed, cluster_state:ok, and every other pod ready and healthy. Healthy means a master owning its shard's slots, or a replica in sync with it.
• Order: replicas first (one that isn't in sync goes before the others), masters last.
• Masters hand over first. A master is never restarted while it is the master:
  • The rollout puts the step-down annotation (kividb.io/step-down) on its pod.
  • The step-down handler asks the shard's most caught-up in-sync replica for CLUSTER FAILOVER. That is the coordinated mode: the master pauses writes and the replica catches up before it wins the election, so no acknowledged write is lost. I checked cmd_failover / run_manual_failover in kividb.
  • The old master is restarted as a replica on a later pass.
  • A request is repeated after 20 s at most, if the engine hasn't finished.
• replicas: 0: the shard has no one to hand over to, so its master restarts in place and its slots are down until it is back. This is documented.
• Stuck pods: a pod that has been unready for 30 s on an old template is replaced straight away, as before.

Sync for shard replicas

• /cluster/status now reports masterLinkUp (master_link_status).
• A replica is in sync when it follows its own shard's master, its link is up, and its offset is within 8 MiB of the master's (the same maxSyncLagBytes as today).
• status.pods[].synced is now set for sharded clusters.

Planned moves are not failovers

When the old master's pod carries the step-down annotation, the move is recorded as a ShardSwitchover Event (Normal). It does not raise a ShardFailover warning and does not set lastFailoverTime. The annotation is removed once the pod is a replica.

KividbDbOps restart

• Sharded clusters are no longer refused.
• The master check uses status.shards[].masterPod.
• An in-sync replica has to be in the master's own shard.
• Between pods, the restart also waits for cluster_state:ok.

Agent

• New endpoint: POST /cluster/failover sends CLUSTER FAILOVER; the engine refuses it on a master. Behind the agent token, like the other endpoints.

Tested

• go build, go vet, go test ./... and helm lint pass.
• The simulated engine now runs CLUSTER FAILOVER (slots and replicas move to the replica), offsets and the link state. Tests in shard_rollout_test.go:
  • 2 shards × 1 replica:
    • one pod at a time, both replicas first;
    • no pod is ever deleted while it owns slots;
    • at the end each -1 is master and each -0 follows it;
    • ShardSwitchover events, no ShardFailover, no lastFailoverTime;
    • the annotations are cleared.
  • replicas: 0: the masters are restarted one by one.
  • Waits for sync:
    • a replica with its link down is replaced first, and then nothing else moves while it is still down;
    • the same while its offset lags by more than 8 MiB;
    • the rollout carries on once it catches up.
  • The step-down annotation on its own:
    • the shard fails over to the in-sync replica and the status follows;
    • with no replica in sync, nothing happens.
  • KividbDbOps: master detection by shard, an in-sync replica from another shard doesn't count, and the wait for cluster_state:ok.
• Not yet: a real multi-node cluster. That's the AKS e2e PR.

[murtazayusuf]

Merged into master as part of #19 (merge commit 4263894), which carried the whole 0.7.0 stack. Closing as included.