feat(cluster): day-2 sync for hcloud instance types and control-plane replicas

cmd/kubeaid-core/root/cluster/sync/sync.go

@@ -31,9 +31,18 @@ var SyncCmd = &cobra.Command{
 				SkipPRWorkflow: skipPRWorkflow,
 			})
 
+		// ArgoCD reconciles whatever is merged into kubeaid-config, but only `bootstrap`
+		// ever re-renders values-capi-cluster.yaml from general.yaml. So a machineType or
+		// control-plane replicas edit never reaches a provisioned ClusterAPI cluster on its
+		// own : re-render it, and let ArgoCD carry it the rest of the way.
+		case constants.CloudProviderHetzner:
+			core.SyncCAPICluster(cmd.Context(), core.SyncCAPIClusterArgs{
+				SkipPRWorkflow: skipPRWorkflow,
+			})
+
 		default:
 			assert.Assert(cmd.Context(), false, fmt.Sprintf(
-				"'cluster sync' is only needed for the Bare Metal (KubeOne) provider - on %s, merged kubeaid-config changes get reconciled by ArgoCD",
+				"'cluster sync' does not support %s yet - re-run 'cluster bootstrap' to re-render values-capi-cluster.yaml",
 				globals.CloudProviderName,
 			))
 		}

cmd/kubeaid-core/root/config/templates/general.yaml

@@ -58,6 +58,23 @@ cluster:
   # Suitable Kubernetes API configurations will be done for you automatically. And they can be
   # changed using the apiSever struct field.
   enableAuditLogging: True
+  # MachineTemplateRotation names the capi-cluster chart's HCloud MachineTemplates
+  # after a hash of their spec, so that changing a machineType rotates the name.
+  # 
+  # ClusterAPI decides whether a Machine is up to date by comparing the name of
+  # the template it was cloned from against the name its owner references — it
+  # never compares the template's contents. Without rotation, a machineType change
+  # therefore never rolls the cluster: it applies silently to machines created
+  # later, leaving a control plane with mixed instance types.
+  # 
+  # Enabling this on an existing cluster renames the template even when its spec
+  # is unchanged, which rolls the control plane once. Scale to at least 3
+  # control-plane replicas first, so etcd keeps quorum while the machines are
+  # replaced one at a time.
+  # 
+  # Hetzner HCloud only — bare-metal workers are Machine objects, not
+  # MachineDeployments, and their templates keep the fixed legacy names.
+  machineTemplateRotation: False
   # ACMEEmail is the contact email used to register with the ACME
   # CA (Let's Encrypt) when cert-manager's ClusterIssuer is
   # rendered. Required when cluster.keycloak.mode=managed (the

docs/TODO.md

@@ -168,67 +168,29 @@ kubeaid-cli with the offending path, same shape as the parser's
 existing `validate` errors. Defer until we hit the next case from a
 different field; the regions one is fixed at source.
 
-### Day-2 `cluster sync` for cloud instance-type changes (AWS / Azure / hcloud)
-
-Changing a machine `instanceType` (control-plane or a node group) in
-general.yaml has no day-2 propagation for a running cloud cluster.
-`cluster sync` hard-refuses every non-bare-metal provider
-(`cmd/kubeaid-core/root/cluster/sync/sync.go` — "only needed for the Bare
-Metal (KubeOne) provider"), and `cluster upgrade` only patches
-`.global.kubernetes.version` + the machine image
-(`pkg/core/upgrade_cluster.go`, `pkg/cloud/*/cluster_upgrade.go`) — it never
-re-renders `instanceType`. Only `bootstrap` re-renders the full
-`values-capi-cluster.yaml`. So today you re-run bootstrap or hand-edit the
-rendered values file.
-
-Goal: extend `cluster sync` to AWS/Azure/hcloud so an `instanceType` change
-in general.yaml reconciles onto the live cluster — with a kubeone-style
-inline diff + double approval, because this runs on live clusters and rolls
-nodes.
-
-Flow (reuses existing primitives):
-
-1. Re-render config from general.yaml → new `values-capi-cluster.yaml` (+ the
-   verbatim `kubeaid-cli.general.yaml` copy) via `getTemplateValues` +
-   `createOrUpdateNonSecretFiles` (`pkg/core/setup_kubeaid_config.go`).
-2. **Inline diff** of the re-rendered files → confirm (approval #1). NEW —
-   there is no inline diff/plan today; the only diff is the forge PR diff.
-3. Create PR → `git.WaitUntilPRMerged` (approval #2 = the merge).
-4. **Delete+recreate** the affected MachineTemplate(s) with the new
-   `instanceType`. Required because MachineTemplate specs are immutable and
-   the capi-cluster chart (kubeaid repo) names them with FIXED names
-   (`<cluster>-control-plane`, `<cluster>-md-0`, `<cluster>-system` /
-   `-user`) — so a plain ArgoCD sync of a new instanceType is rejected by the
-   API server, and CAPI wouldn't roll anyway (the template ref name is
-   unchanged). Generalize the existing `CloudProvider.UpdateMachineTemplate`
-   (`pkg/cloud/cloud_provider.go:29`; AWS/Azure impls in
-   `pkg/cloud/*/cluster_upgrade.go`) from *image* → *instanceType*. hcloud
-   needs an impl too. This is exactly what `cluster upgrade` already does for
-   the image.
-5. Sync the `capi-cluster` ArgoCD app
-   (`SyncArgoCDApp(constants.ArgoCDAppCapiCluster)`).
-6. **Watch** the KubeadmControlPlane (and MachineDeployments) roll — reuse
-   `renderCAPIStatusTable` / the machine-wait logic (`pkg/utils/kubernetes/
-   clusterapi.go`). A CP-template change makes the KCP auto-roll; an
-   MD-template change needs `clusterctl RolloutRestart` (as upgrade does).
-
-Also add the same inline-diff + double-approval gate to `cluster upgrade` —
-it has the PR-merge gate today but no inline diff.
-
-Open scoping questions (decide before building):
-
-- Both control-plane and worker instance types (CP change → KCP roll, node
-  group change → that MachineDeployment rolls)? Assume yes.
-- v1 = instance-type only, or also fold in the *mutable* changes (replicas,
-  add/remove node groups) that sync cleanly with NO template recreate? Those
-  are the easy, fully-GitOps path (PR + sync + watch, no delete+recreate);
-  instance-type is the one field needing the recreate.
-
-Alternative to the CLI-driven delete+recreate: make the capi-cluster chart
-name MachineTemplates by a spec hash (`-md-0-<hash>`) so a new instanceType
-→ new template name → ArgoCD creates it and CAPI rolls naturally (pure
-GitOps, no client-go surgery). That's a kubeaid-repo chart change affecting
-every cluster's rollout behaviour, so it's a separate PR — noted as the
-cleaner long-term end-state if we're willing to touch the chart.
-
-Design captured 2026-07-08 from a brainstorming session; not yet built.
+### Day-2 `cluster sync` for cloud instance-type changes (AWS / Azure)
+
+hcloud is done — see `docs/specs/2026-07-09-machinetemplate-rotation-day2-sync.md`.
+
+The delete+recreate plan this section used to carry was wrong, and worth recording
+why: ClusterAPI decides a Machine is up to date by comparing the *name* of the
+template it was cloned from against the name its owner references
+(`matchesTemplateClonedFrom` for KCP, `MachineTemplateUpToDate` for MachineDeployments).
+It never compares the template's contents. Recreating a template under its old name
+therefore rolls nothing, and the new instanceType applies silently to machines
+created later. `cluster upgrade` only appeared to work because the Kubernetes version
+change was doing the rolling.
+
+The fix shipped for hcloud is to name MachineTemplates after a hash of their spec, so
+a changed instanceType rotates the name and CAPI rolls normally — pure GitOps, no
+client-go surgery. It is gated on `cluster.machineTemplateRotation` because adopting
+it renames the template once, which costs one control-plane roll.
+
+Remaining: AWS and Azure name their MachineTemplates the same fixed way
+(`charts/aws/templates/AWSMachineTemplate.yaml:4`,
+`charts/azure/templates/AzureMachineTemplate.yaml:4`) and have the same silent no-op.
+`cluster sync` still refuses both providers. Porting the `_helpers.tpl` name/spec
+helpers is mechanical; the CLI side already generalises (`planCapiValuesSync` reads
+only the rendered values, and would need each provider's values paths added).
+
+Also still open: `cluster upgrade` has the PR-merge gate but no inline diff.

docs/config-reference.md

@@ -206,11 +206,11 @@ NOTE : Generally, refer to the KubeadmControlPlane CRD instead of the correspond
 |-------|------|---------|-------------|
 | vmSize | `string` |  |  |
 | diskSizeGB | `uint32` |  |  |
-| minSize | `uint` |  | Minimum number of replicas in the nodegroup.<br> |
-| maxSize | `uint` |  | Maximum number of replicas in the nodegroup.<br> |
 | name | `string` |  | Nodegroup name.<br> |
 | labels | `map[string]string` | [] | Labels that you want to be propagated to each node in the nodegroup.<br><br>Each label should meet one of the following criterias to propagate to each of the nodes :<br><br>  1. Has node-role.kubernetes.io as prefix.<br>  2. Belongs to node-restriction.kubernetes.io domain.<br>  3. Belongs to node.cluster.x-k8s.io domain.<br><br>REFER : https://cluster-api.sigs.k8s.io/developer/architecture/controllers/metadata-propagation#machine.<br> |
 | taints | []`k8s.io/api/core/v1.Taint` | [] | Taints that you want to be propagated to each node in the nodegroup.<br> |
+| minSize | `uint` |  | Minimum number of replicas in the nodegroup.<br> |
+| maxSize | `uint` |  | Maximum number of replicas in the nodegroup.<br> |
 
 ## AzureConfig
 
@@ -353,6 +353,7 @@ REFER : https://docs.kubermatic.com/kubeone/v1.13/references/kubeone-cluster-v1b
 | name | `string` |  | Name of the Kubernetes cluster.<br><br>We don't allow using dots in the cluster name, since it can cause issues with tools like<br>ClusterAPI and Cilium : which use the cluster name to generate other configurations.<br> |
 | k8sVersion | `string` |  | Kubernetes version (>= 1.30.0).<br> |
 | enableAuditLogging | `bool` | True | Whether you would like to enable Kubernetes Audit Logging out of the box.<br>Suitable Kubernetes API configurations will be done for you automatically. And they can be<br>changed using the apiSever struct field.<br> |
+| machineTemplateRotation | `bool` | False | MachineTemplateRotation names the capi-cluster chart's HCloud MachineTemplates<br>after a hash of their spec, so that changing a machineType rotates the name.<br><br>ClusterAPI decides whether a Machine is up to date by comparing the name of<br>the template it was cloned from against the name its owner references — it<br>never compares the template's contents. Without rotation, a machineType change<br>therefore never rolls the cluster: it applies silently to machines created<br>later, leaving a control plane with mixed instance types.<br><br>Enabling this on an existing cluster renames the template even when its spec<br>is unchanged, which rolls the control plane once. Scale to at least 3<br>control-plane replicas first, so etcd keeps quorum while the machines are<br>replaced one at a time.<br><br>Hetzner HCloud only — bare-metal workers are Machine objects, not<br>MachineDeployments, and their templates keep the fixed legacy names.<br> |
 | acmeEmail | `string` |  | ACMEEmail is the contact email used to register with the ACME<br>CA (Let's Encrypt) when cert-manager's ClusterIssuer is<br>rendered. Required when cluster.keycloak.mode=managed (the<br>keycloakx and netbird-mgmt Ingresses both need TLS certs);<br>optional otherwise. Used as Issuer.spec.acme.email.<br> |
 | acmeDNS01 | [`ACMEDNS01Config`](#acmedns01config) |  | ACMEDNS01 switches the rendered ClusterIssuer's solver from<br>the HTTP-01 default to DNS-01. Required for the split-horizon<br>mesh pattern: NetBird-exposed services use real public DNS<br>names (e.g. argocd.staging.acme.com) that only resolve inside<br>the mesh — Let's Encrypt can never reach them over HTTP, but<br>proves ownership via a TXT record on the public zone instead.<br>Requires cluster.acmeEmail plus the provider credential in<br>secrets.yaml (acme.cloudflareApiToken).<br> |
 | apiServer | [`APIServerConfig`](#apiserverconfig) |  | Configuration options for the Kubernetes API server.<br> |
@@ -456,11 +457,11 @@ We enforce the user to use SSH, for authenticating to the Git server.</p>
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
 | machineType | `string` |  | HCloud machine type.<br>You can browse all available HCloud machine types here : https://hetzner.com/cloud.<br> |
-| minSize | `uint` |  | Minimum number of replicas in the nodegroup.<br> |
-| maxSize | `uint` |  | Maximum number of replicas in the nodegroup.<br> |
 | name | `string` |  | Nodegroup name.<br> |
 | labels | `map[string]string` | [] | Labels that you want to be propagated to each node in the nodegroup.<br><br>Each label should meet one of the following criterias to propagate to each of the nodes :<br><br>  1. Has node-role.kubernetes.io as prefix.<br>  2. Belongs to node-restriction.kubernetes.io domain.<br>  3. Belongs to node.cluster.x-k8s.io domain.<br><br>REFER : https://cluster-api.sigs.k8s.io/developer/architecture/controllers/metadata-propagation#machine.<br> |
 | taints | []`k8s.io/api/core/v1.Taint` | [] | Taints that you want to be propagated to each node in the nodegroup.<br> |
+| minSize | `uint` |  | Minimum number of replicas in the nodegroup.<br> |
+| maxSize | `uint` |  | Maximum number of replicas in the nodegroup.<br> |
 
 ## HCloudConfig