Potential solution for containers that take longer than 15m to build

[thatderek]

Currently, the whole process of building a container is controlled / gated by controller_lambda.py running in an aws lambda. The issue with this is that it means, given the use of a single lambda, the operation will fail for any container that takes longer than 15m to build.

If people did want to build containers that take longer though, it could maybe be done by chaining a few lambda invocations together: an initial build-it lambda and then a set number of waiter lambdas. Like, for instance, if a container took ~45m to build (and what great sadness it would be if that were the case, but I've seen it before) then it could probably be covered by 2 lambda_invocation resources that looked something like:

resource "aws_lambda_invocation" "builder" { 
  [ all the configs ]
 }
 
resource "aws_lambda_invocation" "waiters" { 
  count = 4 # or maybe a var
  env {
    task-arn = aws_lambda.invocation.builder.result.task-arn-to-watch
  }
}

data "aws_ecr_image" { 
  # other configs
  digest = compact(flatten(aws_lambda_invocation.waiters[*].result["image_digest"])][0]
}

And so every time a container got built, 5 functions would run, each with a timeout of 15m: the first one would be responsible for firing off the kaniko builder in ecs and waiting for the rest of the 15m; each subsequent builder would come up, look for the image, and if it didn't exist, wait up to 15m for it to show up. Perhaps the last in the chain would exit non-zero if the image never showed up. The additional cost would be negligible because, should the image be built in a prior stage, the subsequent lambda firings would only go for a few milliseconds per tf-run.

Anyhow, this is the mechanism I thought to solve this issue but I don't really intend to tackle it unless someone chimes in saying they need it. In the mean time, this is a placeholder for me to remember my thinking in the future should someone need it.