scmigrate

scmigrate is a kubectl plugin and rclone-backed helper image for moving selected Kubernetes PVCs from one StorageClass to another without ever intentionally leaving the cluster with only an in-flight copy of the data.

The plugin is built around resumable phases. It writes migration state to PVC, PV, and sync-pod annotations before each cluster mutation that matters. If the process is interrupted, a later run uses those annotations to continue from the last durable point.

Install

Download the archive for your platform from a tagged release. Archive names use the version without the leading v; release tags and runner image tags keep the leading v.

TAG=v0.1.0
VERSION="${TAG#v}"
OS="$(uname -s | tr '[:upper:]' '[:lower:]')"
ARCH="$(uname -m)"
case "$ARCH" in
  x86_64) ARCH=amd64 ;;
  arm64|aarch64) ARCH=arm64 ;;
  *) echo "unsupported architecture: $ARCH" >&2; exit 1 ;;
esac

ARCHIVE="kubectl-scmigrate_${VERSION}_${OS}_${ARCH}.tar.gz"
BASE_URL="https://github.com/laverya/scmigrate/releases/download/${TAG}"

curl -fL -o "$ARCHIVE" "$BASE_URL/$ARCHIVE"
curl -fL -o checksums.txt "$BASE_URL/checksums.txt"
if command -v sha256sum >/dev/null 2>&1; then
  grep "  $ARCHIVE$" checksums.txt | sha256sum -c -
else
  grep "  $ARCHIVE$" checksums.txt | shasum -a 256 -c -
fi

tar -xzf "$ARCHIVE"
mkdir -p ~/.local/bin
install -m 0755 kubectl-scmigrate ~/.local/bin/kubectl-scmigrate
kubectl scmigrate version

For Windows, download the windows_amd64 archive from the same release and put kubectl-scmigrate.exe on PATH.

Tagged releases publish a matching runner image at:

ghcr.io/laverya/scmigrate-runner:v0.1.0

Release binaries default --runner-image to the matching vX.Y.Z image tag, so you normally do not need to build or push a runner image yourself. The release workflow also publishes a bare X.Y.Z tag and latest, but scmigrate run intentionally rejects latest; use a version tag or digest for migrations.

Permissions

scmigrate uses the credentials in your current kubeconfig. The user, group, or service account that runs kubectl scmigrate must be allowed to list namespaces and to list or mutate PVCs, PVs, pods, supported workload controllers, and the migration-managed ConfigMaps used for StatefulSet restore records.

deploy/rbac.yaml provides the required ClusterRole and binds it to the scmigrate-system/scmigrate ServiceAccount. Use it directly only if you will run the plugin with that service account's credentials. For a local kubectl workflow, ask a cluster admin to bind the scmigrate ClusterRole to the actual kube user or group that will run the migration.

To install that template role and service-account binding:

kubectl create namespace scmigrate-system
kubectl apply -f deploy/rbac.yaml

Before running a real migration, verify permissions in the namespaces you plan to touch:

kubectl auth can-i list persistentvolumeclaims --all-namespaces
kubectl auth can-i patch persistentvolumes
kubectl auth can-i create pods -n default
kubectl auth can-i patch deployments.apps -n default

StatefulSet migrations require permission to delete and recreate StatefulSets. DaemonSet migrations require permission to patch DaemonSets and delete affected pods.

Usage

Plan a migration:

kubectl scmigrate plan \
  --namespace default \
  --selector app=postgres \
  --source-storage-class old-sc \
  --target-storage-class new-sc

Run it:

kubectl scmigrate run \
  --namespace default \
  --selector app=postgres \
  --target-storage-class new-sc \
  --yes

Release-versioned plugin builds default --runner-image to the matching runner image tag, for example ghcr.io/laverya/scmigrate-runner:v0.1.0. Dev builds do not guess; pass --runner-image explicitly when kubectl scmigrate version does not show a default runner image.

PVCs can be selected by namespace, label selector, source storage class, and repeatable annotation filters:

kubectl scmigrate plan \
  --all-namespaces \
  --selector 'migration=scmigrate' \
  --annotation scmigrate/approved=true \
  --target-storage-class premium-rwo

Migration Flow

For each matching bound PVC, the plugin:

1. Records the original PVC metadata/spec in annotations.
2. Creates a temporary destination PVC using the target storage class.
3. Starts a rclone sync pod mounting the source read-only and the destination read-write while the workload remains online.
4. Quiesces every active non-terminal pod that is using that PVC by grouping consumers by their owning workload and scaling/deleting each affected parent once.
5. Runs a final rclone sync pass.
6. Sets both the source PV and destination PV reclaim policies to Retain.
7. Deletes the original and temporary PVCs.
8. Clears the destination PV claimRef.
9. Recreates the PVC with the original name, labels, and annotations, explicitly bound to the destination PV.
10. Restores the workload replica count.

The old PV is deliberately left retained. Delete it manually only after the new PVC has been verified.

The destination PVC does not have to bind before the first sync pod is created. That keeps storage classes with WaitForFirstConsumer binding from deadlocking. When exactly one workload pod is using the source PVC, the initial sync pod is pinned to that same node so local or topology-constrained volumes can bind in the same place as the source consumer.

Avoiding Full Service Outages

Final sync requires the PVC to have no active writers. scmigrate determines all non-terminal pods that mount each selected PVC, resolves their owning workload, and quiesces every affected workload before the final sync and cutover. It also rechecks for active consumers immediately before cutover.

For Deployment and standalone ReplicaSet workloads, the plugin scales each affected parent to zero once per PVC and restores the original replica count after cutover. This covers PVCs shared by multiple replicas and PVCs mounted by multiple parent resources.

For StatefulSet workloads using ordinary volumeClaimTemplates, each PVC is usually consumed by one pod. The plugin sorts those PVCs by trailing ordinal in descending order, stores the StatefulSet restore spec in a migration-managed ConfigMap, temporarily orphans the StatefulSet, deletes the consumer pod, and recreates the StatefulSet after cutover. This keeps one StatefulSet pod down at a time. If a StatefulSet shares one PVC across multiple pods, the plugin scales the StatefulSet to zero and restores the original replica count after cutover.

For DaemonSet workloads, the plugin temporarily switches the DaemonSet to OnDelete, adds required node-affinity exclusions for every node with a pod that is consuming the PVC, and deletes those pods. Other DaemonSet pods keep running because the strategy change prevents a rolling update. After cutover, the original affinity and update strategy are restored and the plugin waits for ready DaemonSet pods on the original nodes.

Resume Model

All important objects are marked with annotations under:

scmigrate.laverya.github.com/*

The temporary destination PVC stores the original PVC metadata and workload quiesce record. That means a run can resume even after the original PVC has been deleted but before the final PVC has been recreated.

During cutover, the destination PV is also annotated with the original PVC snapshot before either PVC is deleted. That lets a later run continue even if the process stops after both PVC objects are gone but before the final PVC is created.

Use kubectl scmigrate run ... --yes again with the same selection flags to continue an interrupted migration.

Keep these flags stable when resuming: --namespace or --all-namespaces, --selector, --annotation, --source-storage-class, and --target-storage-class. If a resumable object was prepared for a different target storage class, the plugin stops instead of continuing with mixed target state.

Operational Notes

• The source PVC must be bound.
• Only filesystem PVCs are supported. Block-mode PVCs are rejected because the rclone runner mounts PVCs as filesystems.
• ReadWriteOncePod PVCs are rejected because the live initial sync requires a second pod to mount the source PVC.
• The destination storage class must support the requested access modes, filesystem volume mode, capacity, and any topology required by the source workload.
• PVC selectors are restored on the final PVC. Static PV selectors that only matched the old PV can prevent the final PVC from binding to the new PV.
• The runner image runs rclone as root so it can preserve ownership and mode bits when --metadata is enabled. Namespaces with restricted Pod Security Admission must allow the migration worker pod or run it under a policy that permits the required filesystem operations.
• run requires an explicit --runner-image with a non-latest tag or digest. For release-versioned plugin builds, this defaults to the matching ghcr.io/laverya/scmigrate-runner:<version> tag.
• The default rclone arguments passed after rclone sync are:

--config=/dev/null --links --metadata --create-empty-src-dirs --stats=15s

• --metadata asks rclone's local backend to preserve mode, owner, group, timestamps, and supported metadata such as user extended attributes.
• --links is required for local symlink transfer. Rclone does not preserve source hard-link relationships as rsync -H did.
• --rclone-args is split on whitespace and passed after rclone sync; it is not evaluated by a shell.
• The plugin leaves PV reclaim policies as Retain by default. Pass --restore-reclaim-policy to restore the destination PV's original policy after cutover.
• The old PV is intentionally retained after a successful migration. Remove it only after validating the new PVC and application data.
• --skip-initial-sync means the whole data copy happens during the final outage window.

Supported Workloads

scmigrate can quiesce pods owned by:

• standalone Pods
• Deployments
• standalone ReplicaSets
• StatefulSets
• DaemonSets

Other owners, including Jobs, CronJobs, Argo Rollouts, operator-specific controllers, and custom controllers, are rejected rather than guessed. If an operator, HPA, or other reconciler can recreate writers while a workload is quiesced, pause that reconciler before running the migration.

Failure Recovery

The migration state is durable across reruns. Use the same command and selection flags to resume.

• Before quiesce: rerun the migration; it reuses the prepared destination PVC and sync pod state.
• After quiesce: rerun the migration promptly. Workloads may intentionally remain scaled down or orphaned until restore completes.
• During cutover: rerun the migration. The temporary destination PVC or the destination PV cutover record contains the original PVC snapshot needed to recreate the final PVC.
• After restore: verify the workload, then clean up retained old PVs and any migration-managed ConfigMaps after you no longer need rollback context.

Development

Build the kubectl plugin from source:

make build VERSION=dev
mkdir -p ~/.local/bin
install -m 0755 bin/kubectl-scmigrate ~/.local/bin/kubectl-scmigrate

Source-built binaries may not have a release-version default runner image. For local testing, build a runner image with a non-latest tag, push it to a registry your target cluster can pull from, and pass it to run:

RUNNER_IMAGE=ghcr.io/you/scmigrate-runner:dev-test
make build VERSION=dev
mkdir -p linux/amd64
cp bin/kubectl-scmigrate linux/amd64/kubectl-scmigrate
docker build --build-arg TARGETPLATFORM=linux/amd64 -f Dockerfile.runner -t "$RUNNER_IMAGE" .
docker push "$RUNNER_IMAGE"

kubectl scmigrate run \
  --namespace default \
  --selector app=postgres \
  --target-storage-class new-sc \
  --runner-image "$RUNNER_IMAGE" \
  --yes

Run unit tests:

make test

Run the kind-backed end-to-end tests:

make e2e

The e2e target creates temporary kind clusters, builds and loads the scratch rclone runner image, then verifies real PVC migrations for Deployment, StatefulSet, DaemonSet, shared-PVC, multi-parent, and etcd StatefulSet cases. It requires docker, kind, and kubectl. Set KEEP_E2E_CLUSTER=1 to leave the cluster running after the test.

Releasing

Public releases are created by pushing a v* tag. The release workflow first checks that the tagged commit already has a successful Test workflow run, then GoReleaser publishes:

• kubectl-scmigrate archives for Linux, macOS, and Windows
• checksums.txt
• the ghcr.io/laverya/scmigrate-runner image for linux/amd64 and linux/arm64

git tag v0.1.0
git push origin v0.1.0