feat(pipeline): retrieve prep from archive if it exists by lemeurherve · Pull Request #7079 · jenkinsci/bom

lemeurherve · 2026-07-15T12:45:04Z

This change allows to skip bash prep.sh if there is an archive including the same commit as the one built, gaining about 30 minutes on every subsequent builds using (or set in replay via fixedPrepArchiveName to) the same commit.

Using a copyArtifacts loop over previous builds as I need to retrieve the archive corresponding to the current commit, not necessary the last one:

Build n°1, initial commit: run bash prep.sh and create new archive, archiveArtifacts it, publish other artifacts (like test reports)
Build n°2, replay or rerun: retrieve archive from build n°1, don't create new archive, publish other artifacts
Build n°3, replay or rerun: retrieve archive from build n°1, don't create new archive, publish other artifacts
Build n° 4, new commit: run bash prep.sh and create new archive, archiveArtifacts it, publish other artifacts
Build n°5, replay or rerun: retrieve archive from build n°4, don't create new archive, publish other artifacts
Build n°6, replay or rerun: retrieve archive from build n°4, don't create new archive, publish other artifacts
etc.

Extracted from:

wip: costs reduction #7037

Refs:

Testing done

CI

First build in 34 mins: https://ci.jenkins.io/job/Tools/job/bom/view/change-requests/job/PR-7079/2

Second build in 32 mins with a new commit 4d2e41948de74ee9ac52f8b372df2c109530a3f3, creating a new archive as expected: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7079/3

Third build in 2 mins 24 sec, rerun with "Build now" without any pipeline or code change: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7079/4

Fourth build, using fixedPrepArchiveName from the first build and showing a warning: https://ci.jenkins.io/job/Tools/job/bom/job/PR-7079/7

--- old/Jenkinsfile
+++ new/Jenkinsfile
@@ -9,7 +9,7 @@
 def fullTestLabel = pullRequest.labels.contains('full-test')
 def weeklyTestLabel = pullRequest.labels.contains('weekly-test')
 
-def fixedPrepArchiveName = '' // can be set to a specific prep archive name in case last commits aren't impacting it
+def fixedPrepArchiveName = 'bom-prep-3f5c63efd0ea6c82d270e47fecc5dc7fb24e2bdc.tar.gz' // can be set to a specific prep archive name in case last commits aren't impacting it

Not marking the build as instable in that case as a PR where changes have been made to a pipeline only could still reuse a prep archived from an earlier build.

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests that demonstrate the feature works or the issue is fixed

jglick · 2026-07-15T15:07:38Z

+  if (buildNumber == 1) {
+    echo "[INFO] First build of ${jobName}, no ${archiveName} available yet"
+  } else {
+    // Loop over builds to retrieve the prep archive as previous build can have (only) other archive(s)


Maybe use lastSuccessfulBuild (note that this includes UNSTABLE) rather than looping?

The issue with lastSuccessFulBuild is that I would not be able to retrieve the bom-report from an "unstable" build (one with a marker for example).

FWIW, I got it looping over more than 100 builds in less than a second in #7037, I was surprised how quick it can be.

The issue with lastSuccessFulBuild is that I would not be able to retrieve the bom-report from an "unstable" build

Why? These would match lastSuccessfulBuild (not lastStableBuild).

I got it looping over more than 100 builds in less than a second

Sure, if they are already loaded into memory! The problem is the case that the SoftReferences have been cleared and you are forcing large objects to reread from disk.

Would you consider this a blocker?

I could probably use something like a S3 archive, but that would require some non trivial infra setup as we're trying to avoid write credentials on ci.jenkins.io as it can't be 100% trusted.

And I'm counting later on on this trick to retrieve the bom report across previous builds too.

Would you consider this a blocker?

Probably not, just caught my eye and alarmed me: in general, any code which loops over historical (completed) build records should be a red flag.

During my (numerous) tests I tried several selectors, but as a build could have archived something else, copyArtifact would use that build as ref to find the archive I wanted, archive not always in that build, sometime in earlier ones. Hence this loop.

I may have missed something, but I'd like to start that way if possible. Let me know if you think of an alternative 🙂

loops over historical (completed) build records should be a red flag

I need to do this to be able to rerun multiple builds profiting from the last available prep:

Build n°1, initial commit: run bash prep.sh and create new archive, archiveArtifact it, publish other artifacts (like test reports)

Build n°2, replay or rerun: retrieve archive from build n°1, don't create new archive, publish other artifacts

Build n°3, replay or rerun: retrieve archive from build n°1, ...

Build n° 4, new commit: run bash prep.sh and create new archive, archiveArtifact it, publish other artifacts

Build n°5, replay or rerun: retrieve archive from build n°4, ...

Build n°6, replay or rerun: retrieve archive from build n°4, ...

etc.

lemeurherve · 2026-07-15T15:28:38Z

I'd like to merge this PR and #7077 as soon as possible, as they'll help me quite a lot to pursue my #7037 split (cf #7073, for jenkins-infra/helpdesk#5208) without having to rerun build executing prep.sh on (too) many plugins everytime in multiple PRs, and as this one will result in immediate costs reductions.

lemeurherve

Completed a bit the PR body and added more explanatory comments below.

Merging to benefit from immediate costs reductions and to allow me to iterate more quickly on jenkins-infra/helpdesk#5208 and #7037 split.

I'll address any post-merge suggestion in follow-up pull request(s) if needed.

lemeurherve · 2026-07-15T17:45:47Z

+          if (junit(testResults: '**/target/surefire-reports/TEST-*.xml,**/target/failsafe-reports/TEST-*.xml').failCount > 0) {
+            error 'Some test failures during prep.sh, not going to continue'
+          }


Moved from mavenNode, we want to stop here if there is an error in prep.sh.

lemeurherve · 2026-07-15T17:47:39Z

+            error 'Some test failures during prep.sh, not going to continue'
+          }
+          // Publish incrementals before prep archive preparation to avoid dirty git status
+          infra.prepareToPublishIncrementals()


Moved from the end of "prep" stage as we can prepare incrementals publication as soon as prep.sh finished, and before dirtying the staging area when we create the new prep archive.

lemeurherve · 2026-07-15T17:48:12Z

+      catchError(buildResult: 'SUCCESS', stageResult: 'NOT_BUILT') {
+        error("[INFO] ${prepArchiveName} not found")
+      }
+      return


Mark stage as skipped.

lemeurherve · 2026-07-15T17:50:19Z

+
+  stage('prep') {
+    if (prepFoundInBuildNumber == 0) {
+      withChecks(name: 'Tests', includeStage: true) {


withCheck moved from mavenNode to here and to L182, as those are the parts we want to record test results from. (Also, avoid dangling GitHub status checks when skipping stages inside mavenNode)

lemeurherve · 2026-07-15T17:51:48Z

+def fullTestLabel = pullRequest.labels.contains('full-test')
+def weeklyTestLabel = pullRequest.labels.contains('weekly-test')


Common location for labels retrieval, quite useful when using replays to reproduce/trigger certain type of builds.

Note: hotfixed in 84e4da8

-def fullTestLabel = pullRequest.labels.contains('full-test') -def weeklyTestLabel = pullRequest.labels.contains('weekly-test') +def fullTestLabel +def weeklyTestLabel +if (env.CHANGE_ID) { + fullTestLabel = pullRequest.labels.contains('full-test') + weeklyTestLabel = pullRequest.labels.contains('weekly-test') +}

lemeurherve · 2026-07-17T13:20:23Z

Merging to benefit from immediate costs reductions and to allow me to iterate more quickly on jenkins-infra/helpdesk#5208 and #7037 split.

Rollbacked for now in #7089

Ref:

Decrease BOM costs #7073 (comment)

lemeurherve added enhancement New feature or request internal labels Jul 15, 2026

This was referenced Jul 15, 2026

wip: costs reduction #7037

Draft

Decrease BOM costs #7073

Open

feat(pipeline): retrieve prep from archive if it exists

4d2e419

lemeurherve force-pushed the 7037-01-prep-archive branch from 3f5c63e to 4d2e419 Compare July 15, 2026 14:09

lemeurherve marked this pull request as ready for review July 15, 2026 14:09

lemeurherve requested a review from a team as a code owner July 15, 2026 14:09

jglick reviewed Jul 15, 2026

View reviewed changes

lemeurherve added 2 commits July 15, 2026 17:21

warn in the stage for visibility

cc7a898

cleanup

26e8975

lemeurherve commented Jul 15, 2026

View reviewed changes

lemeurherve merged commit 4062d3a into jenkinsci:master Jul 15, 2026
2 checks passed

lemeurherve deleted the 7037-01-prep-archive branch July 15, 2026 18:11

lemeurherve added a commit that referenced this pull request Jul 15, 2026

hotfix: pullRequest not available on master (fixup of #7079)

84e4da8

		def fullTestLabel = pullRequest.labels.contains('full-test')
		def weeklyTestLabel = pullRequest.labels.contains('weekly-test')

Uh oh!

Conversation

lemeurherve commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing done

Submitter checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemeurherve Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemeurherve Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemeurherve commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lemeurherve left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lemeurherve Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lemeurherve commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lemeurherve commented Jul 15, 2026 •

edited

Loading

lemeurherve Jul 15, 2026 •

edited

Loading

lemeurherve Jul 15, 2026 •

edited

Loading

lemeurherve commented Jul 15, 2026 •

edited

Loading

lemeurherve Jul 15, 2026 •

edited

Loading

lemeurherve commented Jul 17, 2026 •

edited

Loading