Add CI/CD pipeline and release procedure

[nic-hartley]

BLOCKED until #5 is done

Goal

Ensure that things get to production with a minimum of effort, while still requiring a minimum of quality.

Description

My idea is divided into two stages: Continuous integration with testing, and continuous deployment, in that order. The point is to separate the quality assurance from the actual release of a new version, to avoid confusing the two, and so that we can be sure that a new version will actually be deployed when we trigger the CD -- the CI contains all of the QA checks, so once that's completed successfully, we know that the CD can only fail because the deployment itself failed.

CI

• Automatically listen for new pull requests on master
• Pull and compile the latest code
  • Including any unit tests -- this should be done with a plain mvn clean package, no -DskipTests=true.
• Run any migrations necessary (to make sure those work)
  • Possibly autogenerated from PR?
• Run integration tests (Expand tests #5)
• Compile a changelog
  • How? Maybe reading through the merged PR and looking for a Changelog section? Maybe this has to be written by hand?
• Only after all of those pass is the PR allowed to be merged
  • We might also want to automatically increment the pom.xml version, but I'm not sure how this should best be done.

If any of those steps fails, the rest should be skipped. They're ordered roughly by (a) what naturally needs to come first, and (b) speed -- the quickest stages run first, to fail as quickly as possible.

CD

There are two steps to CD: Staging and prod.

Staging

Staging runs automatically, every time master changes.

• Collects all the statistics from the CI run of the latest PR merged: Tests completed/failed, code coverage, perf results
  • Everything merged in since the last deploy
• Compiles a changelog
  • How? Possibly by reading PRs merged since last deploy?
• Push a new release to GitHub
  • Attach the fat JAR and the new changelog
  • Use the semver number as the tag
• Host the new code on a private test server with the same stack as production
  • Re-run integration tests against it, to make sure things still work
• Fuzz testing?
  • It's extremely helpful but it can be difficult to configure
  • This step may actually eliminate the need for integration tests -- we can assume the code works here, and fuzz tests will catch any non-2xx responses
• Run a perf test against that server

Deploy

The deploy stage requires a human to OK the build and set a time. It's relatively simple:

• Immediately start serving... something that indicates that maintenance will be happening at the specified time
• At the specified time, either entirely shut down the backend or redirect all requests to a static JSON response that says stuff is under maintenance
• Swap out the fat JAR, run migrations on the database and S3, and update the configuration to match the staging server's
• Reenable the backend

Alternatives

• Not having it - A deploy pipeline isn't strictly necessary, but it prevents dumb mistakes (forgetting a semicolon) from taking down prod, ensures that every deployed version meets some minimum standard for quality, and simplifies the prod-deployment process to a single click, so there's no chance of missing a step.
• Different steps - Yeah, this is up for discussion, for sure. At first blush, the steps I outlined seem sufficient, and we can always add or remove them down the line, but if anyone has a different opinion, that's why this is discussion wanted. The steps I outlined are... complex, to say the least. I do think it's important to have a pre-prod staging server, though, which is set up as close as possible to the actual prod stack.

Additional notes

No, a Heroku instance watching master and automatically redeploying does not count.