Merge ValidatesRunner tests into small number of Dataflow jobs, each job hosting multiple tests.

[reuvenlax]

Only jobs in the same test class are eligible for merging; we cannot easily merge jobs in different test classes.

Some types of jobs cannot be merged (e.g. jobs that are expected to throw an exception, which will fail all pipelines hosted in that job). This PR determines which jobs can be so merged, and then merges jobs that are compatible with each other (i.e. they have equivalent PipelineOptions). Currently we only merge batch jobs, though this framework can be extended to streaming as well.

If a merged job fails, we parse the job metrics to see if any of the individual tests have succeeded. We then reexecute the tests for the failed jobs in standalone jobs.

[codecov]

Codecov Report

❌ Patch coverage is 80.85366% with 157 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.17%. Comparing base (76f6e6d) to head (6f2e138).
⚠️ Report is 160 commits behind head on master.

Files with missing lines	Patch %	Lines
...runners/dataflow/DataflowTestBatchCoordinator.java	82.94%	49 Missing and 46 partials ⚠️
...java/org/apache/beam/sdk/testing/TestPipeline.java	72.56%	21 Missing and 10 partials ⚠️
...ache/beam/runners/dataflow/TestDataflowRunner.java	76.47%	18 Missing and 6 partials ⚠️
...che/beam/sdk/testing/BeamParallelJunit4Runner.java	93.02%	1 Missing and 2 partials ⚠️
...g/apache/beam/runners/dataflow/DataflowClient.java	33.33%	1 Missing and 1 partial ⚠️
...apache/beam/sdk/testing/TestPipelineExtension.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40303      +/-   ##
============================================
+ Coverage     56.16%   65.17%   +9.00%     
- Complexity     2288    14125   +11837     
============================================
  Files          1124     1730     +606     
  Lines        178336   109260   -69076     
  Branches       1489    11464    +9975     
============================================
- Hits         100170    71211   -28959     
+ Misses        75645    33775   -41870     
- Partials       2521     4274    +1753     

Flag	Coverage Δ
go	?
java	65.17% <80.85%> (-10.62%)	⬇️
python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[github-actions]

Assigning reviewers:

R: @chamikaramj for label java.
R: @damccorm for label build.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

Converted to draft because it is not ready for review IIUC

[kennknowles]

This is pretty massive, so it will take some time. Is there anything that could help me review it? (other than telling a bot to help me review it)