Allow configuration of app test and build steps

[marlenepereira]

Allow configuration of steps for applications that run unit tests and build artifacts (i.e. binaries) in the build pipeline.

[Niksko]

Any ideas for implementation? It would be awesome to be able to provide additional scripts in the toolbox.yaml that Toolbox would call (with some sort of calling convention, e.g. passing params to the shell script) to generate additional steps at each 'stage' of build.

Stages would be:

• Validation stage - when the current lint/validation steps happen. Would pass the queue to the script called.
• Plan stage - when the current tentative and actual plans happen. Would pass the queue to the script called.
• Apply stage - when the current applies happen. Would pass dev/prod and queue to the script called. You could add additional 'waits' or 'blocks' here if you wanted to.
• Plan and apply additional scripts would be run once for each 'group' of plan and apply steps i.e. run once for dev, once for prod
• Post apply stage - for any stuff you want to do after all of the steps have run.

Does that sound sensible? Thoughts @ashlineldridge @mper0003

The plan and apply extra steps are a little tricky, as a first pass we could just do 'validation' and 'post apply', that would solve some of Edge Networking's problems.

[ashlineldridge]

@Niksko Yeah, this seems to be coming up a bit. When I was writing it, I was imagining that Buildkite usage would fall into one of three main scenarios:

1. You just put make buildkite-pipeline | buildkite-agent pipeline upload as the one and only step in your pipeline.yaml file.
2. You put extra steps before and/or after the make buildkite-pipeline... step.
3. You run make buildkite-pipeline > .buildkite/pipeline.yaml and use the generated pipeline as a starting point for your own fully custom pipeline.

3 would obviously solve all problems but I'm guessing the hesitancy there would be be if, for example, you add a new .tfvars file (e.g., config/production/b.tfvars).

We could invent hooks (e.g., before_validation:, before_apply, etc) in the toolbox.yaml file, but I think that makes it difficult to visualise what the pipeline actually looks like when it gets generated.

What if we created new targets for the individual "stages" - e.g., make buildkite-terraform-plan-pipeline, make buildkite-terraform-apply-pipeline, etc? That way you could compose your own pipeline and slot in the Toolbox "stages" wherever you wanted?