feat(pipeline): split test combinations

[lemeurherve]

This change splits test executions to run many of them from a reasonable number of agents in parallel instead of one per combination.

The default maximum number that I choose to get a baseline is 20, to be refined after more observations.

When it's a weekly-test build on the weekly line, it splits the 295 combinations, to be executed on 20 x 1 parallel agents.
When it's a full-test build on the weekly and oldest lines, it splits the 592 combinations, to be executed on 20 x 2 parallel agents.

Extracted from:

• wip: costs reduction #7037

Tip

Commit to review: e227b2c?w=1

Refs:

• Decrease BOM costs #7073
• Decrease BOM costs by splitting tests more efficiently jenkins-infra/helpdesk#5208 (comment)

Testing done

CI

Initial build, with full-test and limited-plugin-set labels: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/4

Details

Subsequent build, with weekly-test and limited-plugin-set labels + a retry (spot reclaimed): https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/11

Details

Next build, with weekly-test and limited-plugin-set labels + some retries (spot reclaimed): https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/12

Details

---

Build without any label (no weekly-test no full-test), generating a new prep archive and skipping other stages than "flag checks" and "publish incrementals" as expected: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/31/

Build with weekly-test label only, test failure matrix-project (known flake): https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/32

Details

Split combinations:

bom-report_PR-7086_BUILD-32.xml
bom-report_PR-7086_BUILD-32.txt

Build with full-test label only triggered with "Build now" in UI: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7086/32

Details

(in progress)

...

Nodes YAML: https://gist.github.com/lemeurherve/d29ce9c9c65cc9b7af03edc3b7b4cec6

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[timja]

Possible to extract build timings / reliability into a table showing impact?

[lemeurherve]

Possible to extract build timings / reliability into a table showing impact?

Yes, I'm slowing down and taking the time to report everything properly in #7037 and explaining in more details what I've done and integrated¹ before continuing.

Footnotes

1. been going too fast, I really wanted to ship splits before my two PTO weeks, in hope of seeing actual proper costs reductions when I would be back. That was a mistake, presenting my excuses and later probably some rollbacks. ↩

[lemeurherve]

Possible to extract build timings / reliability into a table showing impact?

Started collecting links and data in #7073 comments, more in two weeks.