Description
Add a per-NodeGroup opt-in for spot / low-priority compute across the cluster providers that support it, using the same YAML key (spot: true) everywhere the concept exists. Each provider translates the bool to its own terraform variables; the field itself is provider-native (not part of a shared abstract schema).
Current state
| Provider |
Field |
Behavior today |
| AWS |
Spot bool (yaml:"spot,omitempty") |
Flows through pkg/providers/cluster/aws/tofu.go's untyped node_groups var to the upstream terraform-aws-eks-cluster module. Works but has no godoc, no auto-taint. |
| GCP |
Preemptible bool (yaml:"preemptible,omitempty") |
Legacy naming; dead code (GCP Deploy is a stub, nothing reads the field). Google Cloud's own docs call this "Spot VMs" now; Preemptible is the older generation. |
| Azure |
(no field) |
AKS supports Spot node pools (priority = "Spot", eviction_policy, spot_max_price, plus the kubernetes.azure.com/scalesetpriority=spot:NoSchedule taint), but nothing in NIC exposes it. Every AKS node pool ships as priority = "Regular". |
| Hetzner |
N/A |
Hetzner Cloud has no spot/preemptible tier. Out of scope. |
Proposed changes
- AWS:
Spot bool unchanged. Add godoc. Auto-apply spot=true:NoSchedule taint alongside the existing GPU auto-taint pattern in tofu.go.
- GCP: rename
Preemptible to Spot (YAML preemptible to spot) so all three providers use the same key. GCP is a stub so nothing runtime-breaks; users of the legacy Preemptible YAML would need to update their config once GCP lands for real, but there are no such users today. Google Cloud's own docs recommend Spot as the modern name.
- Azure: add
Spot bool on NodeGroup. In tofu.go, map Spot: true to priority = "Spot" + eviction_policy = "Delete" + spot_max_price = -1 (pay up to on-demand). Auto-apply the AKS spot taint kubernetes.azure.com/scalesetpriority=spot:NoSchedule so general workloads don't schedule onto spot pools by accident.
- Azure: add a NAP-conflict validator. Setting
spot: true on any NodeGroup when cluster.azure.node_provisioning_mode: auto is set should be a hard error with a message pointing at Karpenter's NodePool CRD as the mechanism for per-pool spot control under NAP.
- Hetzner: no change.
Godoc content
Single paragraph, lands in schemas/providers/*.json via PR #507:
Spot enables spot / preemptible capacity for this node group. Spot instances are dramatically cheaper (roughly 60-90% off on-demand) but can be reclaimed by the cloud provider with roughly 30s-2min notice. Only use for stateless workloads that tolerate restart (batch jobs, CI runners, autoscaling web tiers with PodDisruptionBudgets). A spot=true:NoSchedule taint is auto-applied so general workloads don't schedule here; pods that should run on spot must tolerate it explicitly.
Godoc is verbatim across the three providers (with the Azure taint key varying per the cloud convention). Once PR #507 lands, this description flows into schemas/providers/{aws,azure,gcp}.json and the Docusaurus renderer displays it identically per provider.
Explicitly out of scope
Two adjacent consistency gaps surfaced during design but excluded from this issue to keep the change focused:
- Three incompatible taint encodings across providers (AWS
Taint struct with NO_SCHEDULE, GCP Taint struct with NoSchedule casing, Azure []string in "key=value:Effect" form).
- Scaling model divergence (
MinNodes/MaxNodes scalar fields on AWS/GCP/Azure vs Hetzner's Count + optional Autoscaling{Min,Max} struct).
Both are worth doing but not here.
Related
Acceptance criteria
Description
Add a per-
NodeGroupopt-in for spot / low-priority compute across the cluster providers that support it, using the same YAML key (spot: true) everywhere the concept exists. Each provider translates the bool to its own terraform variables; the field itself is provider-native (not part of a shared abstract schema).Current state
Spot bool(yaml:"spot,omitempty")pkg/providers/cluster/aws/tofu.go's untypednode_groupsvar to the upstreamterraform-aws-eks-clustermodule. Works but has no godoc, no auto-taint.Preemptible bool(yaml:"preemptible,omitempty")priority = "Spot",eviction_policy,spot_max_price, plus thekubernetes.azure.com/scalesetpriority=spot:NoScheduletaint), but nothing in NIC exposes it. Every AKS node pool ships aspriority = "Regular".Proposed changes
Spot boolunchanged. Add godoc. Auto-applyspot=true:NoScheduletaint alongside the existing GPU auto-taint pattern intofu.go.PreemptibletoSpot(YAMLpreemptibletospot) so all three providers use the same key. GCP is a stub so nothing runtime-breaks; users of the legacy Preemptible YAML would need to update their config once GCP lands for real, but there are no such users today. Google Cloud's own docs recommend Spot as the modern name.Spot boolonNodeGroup. Intofu.go, mapSpot: truetopriority = "Spot"+eviction_policy = "Delete"+spot_max_price = -1(pay up to on-demand). Auto-apply the AKS spot taintkubernetes.azure.com/scalesetpriority=spot:NoScheduleso general workloads don't schedule onto spot pools by accident.spot: trueon any NodeGroup whencluster.azure.node_provisioning_mode: autois set should be a hard error with a message pointing at Karpenter'sNodePoolCRD as the mechanism for per-pool spot control under NAP.Godoc content
Single paragraph, lands in
schemas/providers/*.jsonvia PR #507:Godoc is verbatim across the three providers (with the Azure taint key varying per the cloud convention). Once PR #507 lands, this description flows into
schemas/providers/{aws,azure,gcp}.jsonand the Docusaurus renderer displays it identically per provider.Explicitly out of scope
Two adjacent consistency gaps surfaced during design but excluded from this issue to keep the change focused:
Taintstruct withNO_SCHEDULE, GCPTaintstruct withNoSchedulecasing, Azure[]stringin"key=value:Effect"form).MinNodes/MaxNodesscalar fields on AWS/GCP/Azure vs Hetzner'sCount+ optionalAutoscaling{Min,Max}struct).Both are worth doing but not here.
Related
descriptionin the generated JSON Schemas automatically.nic configCLI discussion; if/when that lands,nic config init --spotbecomes a natural extension once this issue is closed.Acceptance criteria
spot: trueon an AWS NodeGroup deploys as EKScapacity_type = SPOT(no regression from current behavior) with the auto-taint applied.spot: trueon an Azure NodeGroup deploys an AKS pool withpriority = Spot+ the auto-taint.spot: trueon an Azure NodeGroup withnode_provisioning_mode: autofails atnic validatewith the pointer-to-Karpenter message.Preemptiblerenamed toSpot; YAML keypreemptibletospot. (Runtime behavior unchanged since GCP is a stub.)descriptioninschemas/providers/*.jsonafter PR feat(schema): generate config JSON Schema from the provider registry #507 merges.