feat(cluster): day-2 sync for hcloud instance types and control-plane replicas#20
Draft
ashish1099 wants to merge 1 commit into
Draft
feat(cluster): day-2 sync for hcloud instance types and control-plane replicas#20ashish1099 wants to merge 1 commit into
ashish1099 wants to merge 1 commit into
Conversation
… replicas
Editing machineType or controlPlane.hcloud.replicas in general.yaml had no effect
on a running cluster. Only `bootstrap` re-renders values-capi-cluster.yaml;
`cluster sync` refused every non-bare-metal provider, and `cluster upgrade` only
patched the Kubernetes version and the machine image.
`cluster sync` now handles Hetzner: re-render values-capi-cluster.yaml (plus the
verbatim general.yaml copy), show an inline unified diff, open the PR, sync the
capi-cluster ArgoCD app, and wait for the control plane to converge.
Nothing here deletes and recreates a MachineTemplate, which is what the design in
docs/TODO.md originally called for. ClusterAPI decides a Machine is up to date by
comparing the *name* of the template it was cloned from against the name its owner
references -- never the template's contents. Recreating a template under its old
name therefore rolls nothing, and the new instance type applies silently to
machines created later. The capi-cluster chart instead names templates after a
hash of their spec, so a machineType change rotates the name and CAPI rolls
normally, through plain GitOps. Requires the matching kubeaid chart change, opted
into per-cluster via cluster.machineTemplateRotation.
`cluster upgrade` skips UpdateMachineTemplate on rotation-enabled clusters -- the
fixed-name object it deletes does not exist there -- and syncs the whole app so
the newly-named template lands.
Both commands now confirm which live cluster they are about to touch. Nothing in
kubeaid-cli ever selects a kubeconfig context: every client is built with
clientcmd.BuildConfigFromFlags("", path) and silently follows whatever
current-context the kubeconfig carries. The gate prints the cluster name from
general.yaml, the resolved context, API server and kubeconfig path, plus a loud
MISMATCH line when they disagree. Declining, a missing kubeconfig or no TTY all
abort.
planCapiValuesSync then refuses two changes outright, because both fail silently:
- a rotating change while machineTemplateRotation is false (nothing would roll)
- a rotating change combined with a replicas change (asks CAPI to both add and
replace control-plane members, walking etcd through membership changes with
no quorum to lose)
and warns when a rolling change would run against fewer than 3 replicas.
For netbird-obmondo-com (1 x cpx32 -> 3 x cpx22) that forces the correct order:
scale to 3 first on the existing type, where the template spec and therefore its
name are untouched and the new members simply join etcd; then re-type, where the
rotation rolls the three machines one at a time with quorum held throughout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the "Day-2
cluster syncfor cloud instance-type changes" item indocs/TODO.md, for hcloud.Why
Editing
machineTypeorcontrolPlane.hcloud.replicasin general.yaml had no effect on a running cluster. Onlybootstrapre-rendersvalues-capi-cluster.yaml;cluster syncrefused every non-bare-metal provider, andcluster upgradeonly patched.global.kubernetes.versionplus the machine image.The design changed: rotation, not delete+recreate
The TODO's plan was to delete the immutable MachineTemplate and recreate it with the new
instanceType. That does not work, and fails silently.ClusterAPI decides a Machine is up to date by comparing the name of the template it was cloned from against the name its owner references — never the template's contents (
filters.go:143for KCP,mdutil/util.go:408for MachineDeployments,cluster-api v1.11.10). Recreating under the old name leavesclonedFromNamematching: nothing rolls, and the new instance type applies only to machines created later.The chart now names templates after a hash of their spec (Obmondo/KubeAid#TBD), so:
replicas1 → 3 — spec untouched → name untouched → KCP scales out. New members join etcd; nothing is replaced.machineTypechange — name rotates → KCP rolls, surge then remove.Pure GitOps. No client-go surgery, no window where the template is missing.
What's here
cluster.machineTemplateRotation→global.machineTemplateRotationin the rendered values.cluster synchandles Hetzner: re-render → inline unified diff → confirm → PR → sync the capi-cluster ArgoCD app →WaitForControlPlaneRolloutComplete.cluster upgradeskipsUpdateMachineTemplateon rotation-enabled clusters (the fixed-name object doesn't exist there) and syncs the whole app instead.Gates
Nothing in kubeaid-cli ever selects a kubeconfig context — every client uses
clientcmd.BuildConfigFromFlags("", path)and silently follows whatevercurrent-contextthe kubeconfig carries. Socluster syncandcluster upgradenow resolve that context and make the operator confirm it: cluster name from general.yaml, context, API server, kubeconfig path, and a loudMISMATCHline when they disagree. Declining, a missing kubeconfig, or no TTY all abort.planCapiValuesSyncrefuses two changes outright, because both fail silently:machineTemplateRotationis false — nothing would roll;…and warns when a rolling change would run against fewer than 3 replicas.
Applying to
netbird-obmondo-com(1 × cpx32 → 3 × cpx22)The refusals force the correct order, in two syncs:
replicas: 3, still cpx32, rotation off. Template name unchanged → two new control-plane members join etcd, nothing replaced.machineTemplateRotation: true+machineType: cpx22. Template rotates → the three machines roll one at a time, quorum held.Doing both at once is refused. (cpx22 is a downsize — worth confirming it's big enough for etcd before phase 2.)
Full write-up:
docs/specs/2026-07-09-machinetemplate-rotation-day2-sync.md.Testing
go build,go vet,golangci-lint run ./pkg/core/...(0 issues), and the full suite pass. New table-driven tests cover the refusal/warning matrix and the netbird two-phase sequence, plus kubeconfig-context resolution and the rendered flag.pkg/cloud/hetznerhas a pre-existing failure onmain(TestCreateNetwork/network_already_existsnil-deref panic), untouched by this change.Depends on
Obmondo/KubeAid
feat/machinetemplate-rotation— without it,machineTemplateRotation: truerenders a value the chart ignores, andcluster synccorrectly refuses every instance-type change.🤖 Generated with Claude Code