[murtazayusuf]

Stacked on #16 (then #14, #13, #12, #11, #10). The diff is against #16.

This is the seventh step of 0.7.0: spec.shards can change on a running cluster. Until now the operator created StatefulSets for new shards but never gave them any slots, and on a scale-down it would have deleted shards that still owned slots.

How it goes

1. Start. status.cluster.shards records the count the slots are spread over: set when the cluster forms, or derived from the shards that own slots. When spec.shards differs from it, the operator starts status.cluster.rebalance with from and to. From then on, the shard StatefulSets, PDBs and Services exist for the larger of the two (shardCount), while planning uses spec.shards (targetShards).

2. AddingShards. The rebalance waits until:

   • every pod is ready and known to the anchor;
   • cluster_state:ok;
   • every shard that is to own slots has a master.

   A new shard's pods are all empty masters. Its lowest one is taken as the shard's master, and the others REPLICATE it. This applies only to shards added by the change; elsewhere an empty master is a pod that lost its volume and must keep following.

3. MovingSlots. Ownership is read from the anchor's view, and planMoves computes the moves:

   • each shard over its share gives its highest slots to the shards under theirs, lowest shard first;
   • only what has to move moves: 3→4 moves 4,096 slots, and 4→3 moves shard 3's 4,096;
   • there is one move per (source, destination) pair, since an import takes slots from one source only.

   Each move is a CLUSTER MIGRATION IMPORT on the destination master, one at a time, polled with CLUSTER MIGRATION STATUS:

   • completed: the next move starts;
   • failed, canceled, or the task gone because the destination restarted: the slots stayed with the source. The operator records lastError, sends a SlotMoveFailed warning and retries after 30 s.

4. Done. When the plan is empty:

   • removed shards' StatefulSets and PDBs are deleted, and their Services are cleaned up as usual;
   • their node IDs go into staleNodeIDs, the existing forget loop, until no node knows them;
   • status.cluster.shards = to, and the rebalance is cleared.

   Removed shards' PVCs are kept.

Changing it again half way: from becomes max(from, old to), so a shard that may already own slots is kept until it has given them away.

Meanwhile:

• Rolling restarts wait until the change is done; stuck pods are still replaced.
• Phase Rebalancing, with the move under way and the last error in the Ready message.
• status.cluster.rebalance holds current, up to 16 pending moves, movesDone, lastError and lastErrorTime.

API types, deepcopy and both CRD copies are updated.

Agent

• New endpoints, both behind the token:
  • POST /cluster/migration/import → {taskID};
  • GET /cluster/migration/status?id= → the task, or 404 when the node has no such task.
• RESP2 handling: the engine's map reply arrives as a flat array, which parseMigrationTask reads.

Tested

• go build, go vet, go test ./... and helm lint pass.
• planMoves:
  • 3→4, 4→3, 1→2, 2→8, 8→1 and 3→3;
  • the exact number of slots moved;
  • every shard ends with its share;
  • every move takes slots from their current owner;
  • a second plan is empty.
• The simulated engine: now does CLUSTER MIGRATION IMPORT (one source only, like the real one), keeps tasks, and lists departed nodes as fail the way a real node does.
  • Add a shard (3×1 → 4×1): 4,096 slots moved, the new shard's replica follows its master, phase back to Running.
  • Remove a shard (3×1 → 2×1): 5,461 slots moved, shard 2's pods gone, and every remaining node forgot both removed nodes.
  • A failed move: the error is recorded, there's no retry before 30 s, and the change then completes.
  • Changed half way: 2→4, then 3, keeps 4 shards until shard 3 is drained.
• Against real kividb 1.1.2 (two containers in cluster mode on a Docker network, with the real agent binary):
  • /cluster/migration/import of 8192-16383 completed;
  • 8 keys in slot 16287 moved;
  • the source went to DBSIZE 0;
  • slots 0-8191 / 8192-16383;
  • /cluster/migration/status parsed the task, and an unknown ID gives a 404;
  • an import of slots the node already owns, or from two sources, returns the engine's error.
• Not yet: a reshard under load on Kubernetes. That's the AKS e2e PR.

Known limitation

kividbio/kividb#108: if the destination restarts during a move, the keys it had received stay on it. The slots stay with the source and the move is retried, so the data isn't lost, but those keys take up memory on the destination. This is documented.

[murtazayusuf]

Merged into master as part of #19 (merge commit 4263894), which carried the whole 0.7.0 stack. Closing as included.