Check result of prospective merge in CircleCI

[timobrembeck]

Motivation

At the moment, the CircleCI tests only check the state of a specific branch. It does not check whether the result of a merged PR is valid as well. This is bad because sometimes, there is no actual file conflicts which git detects, but an implicit conflict of e.g. the migration files, or side-effects of tests etc.

Proposed Solution

Add a new job to circleci to merge the changes into the develop branch (locally on the CI server without pushing yet) and then re-run all other jobs to make sure that everything works both on the branch and also after merging.

Unfortunately, there is no straight forward way for doing this in CircleCI, see e.g. this discussions:

• https://discuss.circleci.com/t/show-test-results-for-prospective-merge-of-a-github-pr/1662
• https://discuss.circleci.com/t/checkout-merged-code-for-pull-requests/15398
• https://discuss.circleci.com/t/ability-to-checkout-merge-commits-not-just-branch-commits-maybe-even-make-it-the-default/28777
• https://ideas.circleci.com/cloud-feature-requests/p/show-test-results-for-prospective-merge-of-a-github-pr
• https://ideas.circleci.com/cloud-feature-requests/p/provide-env-variable-for-branch-name-targeted-by-pull-request
• https://gist.github.com/sjones6/396d301543eb026992728cd380344c06

Alternatives

We could also require PRs to be up to date with the default branch before they can be merged. This would add a little bit of overhead for PR authors and require even more rebases than right now.

[timobrembeck]

I think a single job to re-run all other tests is probably a bad idea - it would double our current credit usage... Maybe we can add an initial step of the following kind:

• Check if the current workflow belongs to a PR
  • If yes:
    • Check if there are git conflicts
      • If yes: Run the tests on the specific branch only
      • If no: Merge the PR locally and run the tests on the merged state
  • If no: Run the tests on the specific branch only