Skip to content

Add spot-instance support to cluster provider node groups (Azure), unify naming (rename GCP Preemptible to Spot) #508

Description

@viniciusdc

Description

Add a per-NodeGroup opt-in for spot / low-priority compute across the cluster providers that support it, using the same YAML key (spot: true) everywhere the concept exists. Each provider translates the bool to its own terraform variables; the field itself is provider-native (not part of a shared abstract schema).

Current state

Provider Field Behavior today
AWS Spot bool (yaml:"spot,omitempty") Flows through pkg/providers/cluster/aws/tofu.go's untyped node_groups var to the upstream terraform-aws-eks-cluster module. Works but has no godoc, no auto-taint.
GCP Preemptible bool (yaml:"preemptible,omitempty") Legacy naming; dead code (GCP Deploy is a stub, nothing reads the field). Google Cloud's own docs call this "Spot VMs" now; Preemptible is the older generation.
Azure (no field) AKS supports Spot node pools (priority = "Spot", eviction_policy, spot_max_price, plus the kubernetes.azure.com/scalesetpriority=spot:NoSchedule taint), but nothing in NIC exposes it. Every AKS node pool ships as priority = "Regular".
Hetzner N/A Hetzner Cloud has no spot/preemptible tier. Out of scope.

Proposed changes

  • AWS: Spot bool unchanged. Add godoc. Auto-apply spot=true:NoSchedule taint alongside the existing GPU auto-taint pattern in tofu.go.
  • GCP: rename Preemptible to Spot (YAML preemptible to spot) so all three providers use the same key. GCP is a stub so nothing runtime-breaks; users of the legacy Preemptible YAML would need to update their config once GCP lands for real, but there are no such users today. Google Cloud's own docs recommend Spot as the modern name.
  • Azure: add Spot bool on NodeGroup. In tofu.go, map Spot: true to priority = "Spot" + eviction_policy = "Delete" + spot_max_price = -1 (pay up to on-demand). Auto-apply the AKS spot taint kubernetes.azure.com/scalesetpriority=spot:NoSchedule so general workloads don't schedule onto spot pools by accident.
  • Azure: add a NAP-conflict validator. Setting spot: true on any NodeGroup when cluster.azure.node_provisioning_mode: auto is set should be a hard error with a message pointing at Karpenter's NodePool CRD as the mechanism for per-pool spot control under NAP.
  • Hetzner: no change.

Godoc content

Single paragraph, lands in schemas/providers/*.json via PR #507:

Spot enables spot / preemptible capacity for this node group. Spot instances are dramatically cheaper (roughly 60-90% off on-demand) but can be reclaimed by the cloud provider with roughly 30s-2min notice. Only use for stateless workloads that tolerate restart (batch jobs, CI runners, autoscaling web tiers with PodDisruptionBudgets). A spot=true:NoSchedule taint is auto-applied so general workloads don't schedule here; pods that should run on spot must tolerate it explicitly.

Godoc is verbatim across the three providers (with the Azure taint key varying per the cloud convention). Once PR #507 lands, this description flows into schemas/providers/{aws,azure,gcp}.json and the Docusaurus renderer displays it identically per provider.

Explicitly out of scope

Two adjacent consistency gaps surfaced during design but excluded from this issue to keep the change focused:

  • Three incompatible taint encodings across providers (AWS Taint struct with NO_SCHEDULE, GCP Taint struct with NoSchedule casing, Azure []string in "key=value:Effect" form).
  • Scaling model divergence (MinNodes/MaxNodes scalar fields on AWS/GCP/Azure vs Hetzner's Count + optional Autoscaling{Min,Max} struct).

Both are worth doing but not here.

Related

Acceptance criteria

  • spot: true on an AWS NodeGroup deploys as EKS capacity_type = SPOT (no regression from current behavior) with the auto-taint applied.
  • spot: true on an Azure NodeGroup deploys an AKS pool with priority = Spot + the auto-taint.
  • spot: true on an Azure NodeGroup with node_provisioning_mode: auto fails at nic validate with the pointer-to-Karpenter message.
  • GCP Preemptible renamed to Spot; YAML key preemptible to spot. (Runtime behavior unchanged since GCP is a stub.)
  • Godoc lands verbatim on the three provider fields and appears as description in schemas/providers/*.json after PR feat(schema): generate config JSON Schema from the provider registry #507 merges.
  • Unit tests cover the auto-taint injection on AWS + Azure, and the NAP-conflict validator.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    aws-providerAWS provider related issuesazure-providerAzure provider related issuescloud-providerCloud provider related issues (AWS, GCP, Azure, Local)enhancementNew feature or requestgcp-providerGCP provider related issues

    Fields

    Priority

    None yet

    Start date

    None yet

    Target date

    None yet

    Size

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions