Skip to content

feat(cluster): day-2 sync for hcloud instance types and control-plane replicas#20

Draft
ashish1099 wants to merge 1 commit into
mainfrom
worktree-feat-cluster-sync-instance-type
Draft

feat(cluster): day-2 sync for hcloud instance types and control-plane replicas#20
ashish1099 wants to merge 1 commit into
mainfrom
worktree-feat-cluster-sync-instance-type

Conversation

@ashish1099

Copy link
Copy Markdown
Member

Closes the "Day-2 cluster sync for cloud instance-type changes" item in docs/TODO.md, for hcloud.

Why

Editing machineType or controlPlane.hcloud.replicas in general.yaml had no effect on a running cluster. Only bootstrap re-renders values-capi-cluster.yaml; cluster sync refused every non-bare-metal provider, and cluster upgrade only patched .global.kubernetes.version plus the machine image.

The design changed: rotation, not delete+recreate

The TODO's plan was to delete the immutable MachineTemplate and recreate it with the new instanceType. That does not work, and fails silently.

ClusterAPI decides a Machine is up to date by comparing the name of the template it was cloned from against the name its owner references — never the template's contents (filters.go:143 for KCP, mdutil/util.go:408 for MachineDeployments, cluster-api v1.11.10). Recreating under the old name leaves clonedFromName matching: nothing rolls, and the new instance type applies only to machines created later.

The chart now names templates after a hash of their spec (Obmondo/KubeAid#TBD), so:

  • replicas 1 → 3 — spec untouched → name untouched → KCP scales out. New members join etcd; nothing is replaced.
  • machineType change — name rotates → KCP rolls, surge then remove.

Pure GitOps. No client-go surgery, no window where the template is missing.

What's here

  • cluster.machineTemplateRotationglobal.machineTemplateRotation in the rendered values.
  • cluster sync handles Hetzner: re-render → inline unified diff → confirm → PR → sync the capi-cluster ArgoCD app → WaitForControlPlaneRolloutComplete.
  • cluster upgrade skips UpdateMachineTemplate on rotation-enabled clusters (the fixed-name object doesn't exist there) and syncs the whole app instead.

Gates

Nothing in kubeaid-cli ever selects a kubeconfig context — every client uses clientcmd.BuildConfigFromFlags("", path) and silently follows whatever current-context the kubeconfig carries. So cluster sync and cluster upgrade now resolve that context and make the operator confirm it: cluster name from general.yaml, context, API server, kubeconfig path, and a loud MISMATCH line when they disagree. Declining, a missing kubeconfig, or no TTY all abort.

planCapiValuesSync refuses two changes outright, because both fail silently:

  1. a rotating change while machineTemplateRotation is false — nothing would roll;
  2. a rotating change combined with a replicas change — asks CAPI to both add and replace control-plane members, walking etcd through membership changes with no quorum to lose.

…and warns when a rolling change would run against fewer than 3 replicas.

Applying to netbird-obmondo-com (1 × cpx32 → 3 × cpx22)

The refusals force the correct order, in two syncs:

  1. Scale out. replicas: 3, still cpx32, rotation off. Template name unchanged → two new control-plane members join etcd, nothing replaced.
  2. Re-type. machineTemplateRotation: true + machineType: cpx22. Template rotates → the three machines roll one at a time, quorum held.

Doing both at once is refused. (cpx22 is a downsize — worth confirming it's big enough for etcd before phase 2.)

Full write-up: docs/specs/2026-07-09-machinetemplate-rotation-day2-sync.md.

Testing

go build, go vet, golangci-lint run ./pkg/core/... (0 issues), and the full suite pass. New table-driven tests cover the refusal/warning matrix and the netbird two-phase sequence, plus kubeconfig-context resolution and the rendered flag.

pkg/cloud/hetzner has a pre-existing failure on main (TestCreateNetwork/network_already_exists nil-deref panic), untouched by this change.

Depends on

Obmondo/KubeAid feat/machinetemplate-rotation — without it, machineTemplateRotation: true renders a value the chart ignores, and cluster sync correctly refuses every instance-type change.

🤖 Generated with Claude Code

… replicas

Editing machineType or controlPlane.hcloud.replicas in general.yaml had no effect
on a running cluster. Only `bootstrap` re-renders values-capi-cluster.yaml;
`cluster sync` refused every non-bare-metal provider, and `cluster upgrade` only
patched the Kubernetes version and the machine image.

`cluster sync` now handles Hetzner: re-render values-capi-cluster.yaml (plus the
verbatim general.yaml copy), show an inline unified diff, open the PR, sync the
capi-cluster ArgoCD app, and wait for the control plane to converge.

Nothing here deletes and recreates a MachineTemplate, which is what the design in
docs/TODO.md originally called for. ClusterAPI decides a Machine is up to date by
comparing the *name* of the template it was cloned from against the name its owner
references -- never the template's contents. Recreating a template under its old
name therefore rolls nothing, and the new instance type applies silently to
machines created later. The capi-cluster chart instead names templates after a
hash of their spec, so a machineType change rotates the name and CAPI rolls
normally, through plain GitOps. Requires the matching kubeaid chart change, opted
into per-cluster via cluster.machineTemplateRotation.

`cluster upgrade` skips UpdateMachineTemplate on rotation-enabled clusters -- the
fixed-name object it deletes does not exist there -- and syncs the whole app so
the newly-named template lands.

Both commands now confirm which live cluster they are about to touch. Nothing in
kubeaid-cli ever selects a kubeconfig context: every client is built with
clientcmd.BuildConfigFromFlags("", path) and silently follows whatever
current-context the kubeconfig carries. The gate prints the cluster name from
general.yaml, the resolved context, API server and kubeconfig path, plus a loud
MISMATCH line when they disagree. Declining, a missing kubeconfig or no TTY all
abort.

planCapiValuesSync then refuses two changes outright, because both fail silently:

  - a rotating change while machineTemplateRotation is false (nothing would roll)
  - a rotating change combined with a replicas change (asks CAPI to both add and
    replace control-plane members, walking etcd through membership changes with
    no quorum to lose)

and warns when a rolling change would run against fewer than 3 replicas.

For netbird-obmondo-com (1 x cpx32 -> 3 x cpx22) that forces the correct order:
scale to 3 first on the existing type, where the template spec and therefore its
name are untouched and the new members simply join etcd; then re-type, where the
rotation rolls the three machines one at a time with quorum held throughout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant