Skip to content

fix(aws): serialize scheduled actions per Auto Scaling group #127

Description

@so0k

Summary

Creating several ScheduledActions on one Auto Scaling group fails intermittently. Terraform applies the aws_autoscaling_schedule resources concurrently, and AWS rejects concurrent PutScheduledUpdateGroupAction calls against the same group with:

AlreadyExists: Scheduled action with this scheduled start time already exists

Which action loses varies per run, so a plan that applied cleanly once can fail on the next apply with no configuration change.

Observed in

integ/aws/compute/apps/autoscaling.custom-scaling.ts creates four scheduled actions on one group, none of which set startTime (only Schedule.cron() recurrences):

Run Failing action Recurrence
1 ScaleUpInTheWeekDay 0/10 * * * MON-SUN
2 ScaleUpInTheMorning 0 8 * * *

The victim changes between runs while the configuration is identical, so this is concurrency-dependent, not recurrence-dependent. Note the four recurrences resolve to four distinct next-occurrence times in both runs, so "two crons collided" does not explain it.

Serializing the applies fixes it

Applying the same configuration with -parallelism=1 created all four actions cleanly:

aws_autoscaling_schedule...ScaleUpInTheMorning:   Creation complete after 1s
aws_autoscaling_schedule...ScaleUpInTheWeekDay:   Creation complete after 1s
aws_autoscaling_schedule...ScaleUpInTheDay:       Creation complete after 0s
aws_autoscaling_schedule...ScaleDownAtNight:      Creation complete after 1s
Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

Two of those completed inside the same wall-clock second and still succeeded. So AWS rejects concurrent in-flight Put calls on one group rather than actions that happen to share a start time — serialization is sufficient, and distinct explicit startTimes are not required.

What the sources say

AWS documents both uniqueness constraints as intended behavior (scheduled scaling, Limitations):

The names of scheduled actions must be unique per Auto Scaling group.
A scheduled action must have a unique time value. If you attempt to schedule an activity at a time when another scaling activity is already scheduled, the call is rejected and returns an error indicating that a scheduled action with this scheduled start time already exists.

What AWS does not document is what StartTime becomes when only Recurrence is supplied. Debug logs in hashicorp/terraform-provider-aws#41200 show AWS returning a concrete StartTime equal to the next cron occurrence, and the provider then persisting it in state — but that is inference from one report, not documentation.

The Terraform registry docs for aws_autoscaling_schedule say nothing about concurrent creation, depends_on, or this error. No provider issue exists for this exact error on this resource. The nearest precedent, #33428 (sibling aws_appautoscaling_scheduled_action, open), has a maintainer confirming the whole family shares one non-atomic Put for create and update; the meta-issue tracking that pattern, #44212, states the planned remediation is "still potentially prone to race conditions" and does not list aws_autoscaling_schedule. One user there tried depends_on chaining and it did not help — different error on a different resource, but worth knowing there is no confirmed upstream precedent for chaining as a fix.

Proposed fix

A synth-time aspect that collects every ScheduledAction per autoScalingGroupName and chains them with node.addDependency in construct-path order, so Terraform creates them sequentially.

Synth-time rather than constructor-time registration, so the ordering is independent of construction order and also covers new ScheduledAction(...) used directly (it is a public export, not only reachable via scaleOnSchedule()). TerraformDependencyAspect (src/stack-base.ts) already translates construct dependencies into depends_on on the L1, so no manual plumbing is needed.

Open questions:

  • Skip chaining when the user supplies an explicit startTime, where a collision cannot arise?
  • Worth filing upstream against hashicorp/terraform-provider-aws, since no issue exists for this error and #44212 is a natural hook for requesting read-before-create for this resource.

Blast radius

  • One source file (src/aws/compute/auto-scaling/scheduled-action.ts), plus the aspect.
  • No existing unit test breaks and no snapshot changes: every current test creates exactly one scheduled action per group, so there is never a predecessor to depend on.
  • New tests needed: multi-action chaining and its order, two groups in one stack not cross-chaining, and the imported-group path.
  • integ/aws/compute/autoscaling_test.go becomes the acceptance test.

Not caused by tagging

Discovered while validating #125 / PR #126. The tagging change does not affect scheduled actions; PR #126 documents this flake in the fixture and points here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions