Skip to content

[DO NOT MERGE]Build tensorflow 2.21.0 and run tests.#10724

Open
akritkbehera wants to merge 4 commits into
cms-sw:IB/CMSSW_20_1_X/masterfrom
akritkbehera:Build_tensorflow_2.21.0
Open

[DO NOT MERGE]Build tensorflow 2.21.0 and run tests.#10724
akritkbehera wants to merge 4 commits into
cms-sw:IB/CMSSW_20_1_X/masterfrom
akritkbehera:Build_tensorflow_2.21.0

Conversation

@akritkbehera

Copy link
Copy Markdown
Contributor

No description provided.

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @akritkbehera for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

@cmsbuild

cmsbuild commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54710/summary.html
COMMIT: 948cb84
CMSSW: CMSSW_20_1_X_2026-07-12-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54710/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc13/external/py3-onnx/1.21.0-c375c743f4e6702e893aaddc8ec3f988/cmsdist-pip-src/onnx-1.21.0
Building wheel for onnx (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for onnx
Failed to build onnx
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.VX1Rsl (%build)

RPM build warnings:
Macro expanded in comment on line 554: %{pkginstroot}/bin/*

Macro expanded in comment on line 559: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}


@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

Comment thread acts.spec
@@ -1,13 +1,13 @@
### RPM external acts v44.0.1
### RPM external acts 46.8.1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, this will break our ACTS/Traccc tests and possible the ongoing developments.

I'm in favour of updating ACTS, but we need to synchronise our work.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can create a separate IBs for this then and track it.

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54727/summary.html
COMMIT: b223515
CMSSW: CMSSW_20_1_X_2026-07-13-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54727/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

libcudacxx_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/cuda/13.3.1-723beba4702fd72a837a16c836f8224b/lib64/cmake/libcudacxx
nlohmann_json_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/json/3.12.0-8ae359b0f8d167e8240a776436a49d72/share/cmake/nlohmann_json
pybind11_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/py3-pybind11/3.0.1-9a0007668b2b31843b6b1eaf989187f9/share/cmake/pybind11
rocprim_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/rocprim/7.2.4-2fba8176b61ff820348920a3cb5aaf98/lib/cmake/rocprim
rocthrust_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/rocthrust/7.2.4-bec48eec9fdf9e0e6f1771602f9b21e5/lib/cmake/rocthrust
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.d2wdaU (%build)

RPM build warnings:
Macro expanded in comment on line 527: %{pkginstroot}/python




@akritkbehera akritkbehera linked an issue Jul 13, 2026 that may be closed by this pull request
@smuzaffar

Copy link
Copy Markdown
Contributor

please test

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54735/summary.html
COMMIT: b223515
CMSSW: CMSSW_20_1_X_2026-07-13-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54735/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

libcudacxx_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/cuda/13.3.1-723beba4702fd72a837a16c836f8224b/lib64/cmake/libcudacxx
nlohmann_json_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/json/3.12.0-8ae359b0f8d167e8240a776436a49d72/share/cmake/nlohmann_json
pybind11_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/py3-pybind11/3.0.1-9a0007668b2b31843b6b1eaf989187f9/share/cmake/pybind11
rocprim_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/rocprim/7.2.4-2fba8176b61ff820348920a3cb5aaf98/lib/cmake/rocprim
rocthrust_DIR:PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/rocthrust/7.2.4-bec48eec9fdf9e0e6f1771602f9b21e5/lib/cmake/rocthrust
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.iEZduE (%build)

RPM build warnings:
Macro expanded in comment on line 539: %{pkginstroot}/python




@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54808/summary.html
COMMIT: bf2fe0a
CMSSW: CMSSW_20_1_X_2026-07-17-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54808/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 141 lines to the logs
  • Reco comparison results: 16052 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3548294
  • DQMHistoTests: Total failures: 15230
  • DQMHistoTests: Total nulls: 6
  • DQMHistoTests: Total successes: 3533040
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.032 KiB( 44 files compared)
  • DQMHistoSizes: changed ( 2024.0020001 ): -0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0030001 ): 0.016 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2025.0000002 ): 0.020 KiB JetMET/SUSYDQM
  • Checked 195 log files, 163 edm output root files, 45 DQM output files
  • TriggerResults: no differences found

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 21 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step2 max memory diff 50.9 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step3 max memory diff 42.6 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 51.1 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 51.3 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 41.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 44.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 43.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 41.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 45.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 44.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 43.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 41.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 42.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 41.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 42.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 41.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 42.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 45.4 exceeds +/- 30.0 MiB

@smuzaffar

Copy link
Copy Markdown
Contributor

please test with cms-sw/cmssw#51507

@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54809/summary.html
COMMIT: bf2fe0a
CMSSW: CMSSW_20_1_X_2026-07-17-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54809/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 149 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 16051 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3548294
  • DQMHistoTests: Total failures: 15230
  • DQMHistoTests: Total nulls: 6
  • DQMHistoTests: Total successes: 3533040
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.032 KiB( 44 files compared)
  • DQMHistoSizes: changed ( 2024.0020001 ): -0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0030001 ): 0.016 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2025.0000002 ): 0.020 KiB JetMET/SUSYDQM
  • Checked 195 log files, 163 edm output root files, 45 DQM output files
  • TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34434.404 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially removed 42 lines from the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 113 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167683
  • DQMHistoTests: Total failures: 11492
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 156191
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 24 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

NVIDIA_T4 Comparison Summary

Summary:

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 20 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step3 max memory diff 41.2 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step2 max memory diff 49.6 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 39.5 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 50.1 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 43.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 41.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 44.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 43.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 41.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 38.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 38.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 39.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 43.6 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 54.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 54.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 54.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step3 max memory diff 48.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 61.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 54.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 54.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 54.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 56.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 57.0 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_L40S

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 54.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 53.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step3 max memory diff 48.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 53.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 61.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 53.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 53.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 53.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 330.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 55.5 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 11 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 54.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 53.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 53.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 53.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 61.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 53.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 53.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 257.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 56.0 exceeds +/- 30.0 MiB

@smuzaffar

Copy link
Copy Markdown
Contributor

please test with cms-sw/cmssw#51507 for el9_arrch64_gcc13

@akritkbehera

akritkbehera commented Jul 18, 2026

Copy link
Copy Markdown
Contributor Author

@smuzaffar Out of curiosity what does flags FORCE_LINK="1"/ do?

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54811/summary.html
COMMIT: bf2fe0a
CMSSW: CMSSW_20_1_X_2026-07-16-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10724/54811/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

CMSBUILD_PACKAGE=flatbuffers;  export _CMSBUILD_BUILD_ENV_=1; export TMPDIR=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef/cmsdist-tmp;  source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/rpm-env.sh; source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/pkgtools/build-system-env.sh 20;  rpmbuild  --noclean --buildroot /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/924cdd667c776fd73e00352b00b243ef -bc --short-circuit --define '_smp_build_nthreads 20' --define '_topdir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir' --define "compiling_processes 20"  --define '_builddir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef' --define "cmsdist_directory /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cmsdist" --define "compilerv 1340" --define "cmscompilerv 13" --define "cmsos el9_aarch64" --define "__bootstrap ~bootstrap" --define "almalinux_ver 9" --define "almalinux 9" --define "centos_ver 9" --define "centos 9" --define "rhel 9" --define "dist %{!?distprefix0:%{?distprefix}}%{expand:%{lua:for i=0,9999 do print("%{?distprefix" .. i .."}") end}}.el9%{?with_bootstrap:~bootstrap}" --define "el9 1" --define "dist_vendor         AlmaLinux OS Foundation" --define "dist_name           AlmaLinux" --define "dist_home_url       https://almalinux.org/" --define "dist_bug_report_url https://bugs.almalinux.org/" --define "package_vectorization %{nil}" --define "cmsswdata_version_link 1" --define "archfirst yes" --define "cmsBuild_bootstrap 1"   /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/SPECS/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef/spec >>/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef/log 2>&1
Checking package dependencies: external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20
Done checking package dependencies: external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20
Checking local path dependency for rpm package external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20 just build.
RPM installation stderr bazel:
error: Failed dependencies:
	libc.so.6(GLIBC_2.10)(64bit) is needed by external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20-1-1.aarch64
	libc.so.6(GLIBC_2.11)(64bit) is needed by external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20-1-1.aarch64
	libc.so.6(GLIBC_2.14)(64bit) is needed by external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20-1-1.aarch64
	libc.so.6(GLIBC_2.15)(64bit) is needed by external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20-1-1.aarch64
	libc.so.6(GLIBC_2.16)(64bit) is needed by external+bazel+7.7.0-78d5fb02cddf6c39906373f7af839f20-1-1.aarch64


@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_amd64_gcc13

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

Comment thread bazel.spec Outdated
## INCLUDE cpp-standard
Source: https://github.com/bazelbuild/bazel/releases/download/%{realversion}/bazel-%{realversion}-linux-x86_64
BuildRequires: java-env
AutoReqProv: no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akritkbehera , please remove this and fix the source for aarch64 ( use different soruce for amd and aarch64).

@smuzaffar

Copy link
Copy Markdown
Contributor

@akritkbehera , please also go through the comments I had posted a week ago

@akritkbehera

Copy link
Copy Markdown
Contributor Author

Ah I have force pushed the wrong changes to the branch.

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54892/summary.html
COMMIT: 6c58490
CMSSW: CMSSW_20_1_X_2026-07-22-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10724/54892/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib/python3.12/subprocess.py", line 1026, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib/python3.12/subprocess.py", line 1955, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/bazel/7.7.0-ed88e495d4bc4818d2967e3aa5d1d30b/bin/bazel'
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.AvtmGo (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.AvtmGo (%build)



@cmsbuild

Copy link
Copy Markdown
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54889/summary.html
COMMIT: 6c58490
CMSSW: CMSSW_20_1_X_2026-07-23-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10724/54889/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 147 lines to the logs
  • Reco comparison results: 16052 differences found in the comparisons
  • DQMHistoTests: Total files compared: 45
  • DQMHistoTests: Total histograms compared: 3559760
  • DQMHistoTests: Total failures: 15232
  • DQMHistoTests: Total nulls: 6
  • DQMHistoTests: Total successes: 3544504
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.032 KiB( 44 files compared)
  • DQMHistoSizes: changed ( 2024.0020001 ): -0.004 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2024.0030001 ): 0.016 KiB JetMET/SUSYDQM
  • DQMHistoSizes: changed ( 2025.0000002 ): 0.020 KiB JetMET/SUSYDQM
  • Checked 195 log files, 163 edm output root files, 45 DQM output files
  • TriggerResults: found differences in 1 / 44 workflows

AMD_MI300X Comparison Summary

Summary:

AMD_W7900 Comparison Summary

Summary:

NVIDIA_H100 Comparison Summary

Summary:

NVIDIA_L40S Comparison Summary

Summary:

NVIDIA_T4 Comparison Summary

Summary:

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 21 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0000002_RunZeroBias2025B_10k step2 max memory diff 49.6 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0010001_RunJetMET02025C_10k step3 max memory diff 39.4 exceeds +/- 30.0 MiB
  • Error: Workflow 2025.0010001_RunJetMET02025C_10k step2 max memory diff 32.9 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step2 max memory diff 50.1 exceeds +/- 30.0 MiB
  • Error: Workflow 18434.0_TTbar_14TeV+2026 step3 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step2 max memory diff 43.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step5 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.0_TTbar_14TeV+Run4D121 step3 max memory diff 41.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.75_TTbar_14TeV+Run4D121_HLT75e33Timing step2 max memory diff 44.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step3 max memory diff 41.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step2 max memory diff 43.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.911_TTbar_14TeV+Run4D121_DD4hep step5 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 38.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step5 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 38.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step5 max memory diff 40.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step3 max memory diff 41.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 43.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff 42.4 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 54.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 55.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 54.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step3 max memory diff 48.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 62.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 54.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 54.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 54.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 59.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 56.8 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_L40S

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 54.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 54.0 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 53.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step3 max memory diff 48.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 53.9 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 61.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 54.1 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 54.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 57.7 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 56.3 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 12 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step3 max memory diff 54.8 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.402_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka step2 max memory diff 53.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step2 max memory diff 53.4 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.403_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Validation step3 max memory diff 67.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step3 max memory diff 61.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling step2 max memory diff 53.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step3 max memory diff 40.2 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.712_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnly step2 max memory diff 53.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step2 max memory diff 53.6 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.713_TTbar_14TeV+Run4D121_lstmkFitOnGPUIters01TrackingOnlyAlpakaValidationLST step3 max memory diff 40.5 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 57.3 exceeds +/- 30.0 MiB
  • Error: Workflow 34434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka step2 max memory diff 56.8 exceeds +/- 30.0 MiB

Upgrade the TensorFlow build to 2.21.0:
- Update tensorflow-sources with the new upstream tag and Bazel build
  options: 8 parallel jobs, capped build memory, and the jpegxl
  encoding optimization flag.
- Add tensorflow-tf2xla-return-type.patch to fix -Werror=return-type in
  the tf2xla EncodedBufferAllocationInfo header.
- Disable Bazel automatic dependency provisioning and refresh bazel.spec
  for the new build.
- Force-link tensorflow_cc, which was dropped by the as-needed linker
  flag in CMSSW.
- Point cms-tfaot at the tf-2.21.0-aot-metadata-objects branch that
  links the AOT metadata objects.
Align the core C++ dependencies with the versions required by
TensorFlow 2.21.0:
- Refresh the abseil-cpp libraries and add abseil-cpp as an explicit
  dependency of tensorflow-xla-runtime and protobuf.
- Update the protobuf, grpc, eigen, flatbuffers and java-env specs, and
  drop patches no longer needed with the new versions
  (grpc-mno-outline-atomics, protobuf-non-virtual-dtor,
  protobuf_text_format).
- Adjust the tensorflow-xla-runtime spec and its abseil patch.
@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

1 similar comment
@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54894/summary.html
COMMIT: 9bc1a49
CMSSW: CMSSW_20_1_X_2026-07-22-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10724/54894/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

FATAL: malformed spec found while quering it. Command: 
source /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/rpm-env.sh ;  rpm -q --specfile /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/tmpspec-bazel --info --define "cmsdist_directory /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cmsdist" --define "compilerv 1340" --define "cmscompilerv 13" --define "cmsos el9_aarch64" --define "__bootstrap ~bootstrap" --define "almalinux_ver 9" --define "almalinux 9" --define "centos_ver 9" --define "centos 9" --define "rhel 9" --define "dist %{!?distprefix0:%{?distprefix}}%{expand:%{lua:for i=0,9999 do print("%{?distprefix" .. i .."}") end}}.el9%{?with_bootstrap:~bootstrap}" --define "el9 1" --define "dist_vendor         AlmaLinux OS Foundation" --define "dist_name           AlmaLinux" --define "dist_home_url       https://almalinux.org/" --define "dist_bug_report_url https://bugs.almalinux.org/" --define "package_vectorization %{nil}" --define "cmsswdata_version_link 1" --define "archfirst yes" --define "cmsBuild_bootstrap 1"  --define 'buildroot /foo'
Resulted in:

warning: line 36: It's not recommended to have unversioned Obsoletes: Obsoletes: external+bazel+7.7.0-5f0a6f38d977157fbcf8813a3e764519
error: line 546: Unclosed %if
error: query of specfile /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/tmpspec-bazel failed, can't parse
Traceback (most recent call last):
  File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/./pkgtools/cmsBuild", line 5312, in 
    build(opts, args[1:], PKGFactory)
  File "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/./pkgtools/cmsBuild", line 4479, in build


Update packages that failed in CI after the abseil-cpp and TensorFlow
bumps:
- Update the acts (with an ACTS tag bump), fasthadd and
  triton-inference-client specs.
- Update pip packages onnx, torch and requirements, and the
  torch-extension-build helper.
- Add lwtnn-assert-fix.patch and patches/pip/pyg-lib.patch.

Address changes

Fix Bazel for ARM
@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54896/summary.html
COMMIT: 159bb9f
CMSSW: CMSSW_20_1_X_2026-07-22-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10724/54896/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

+ bazel --batch --output_user_root ../build --host_jvm_args=--add-opens=java.base/java.nio=ALL-UNNAMED --host_jvm_args=--add-opens=java.base/java.lang=ALL-UNNAMED build -s --verbose_failures --copt=-march=armv8-a --copt=-mno-outline-atomics --config=opt --cxxopt=-std=c++20 --host_cxxopt=-std=c++20 --jobs=-j 20 --config=nogcp --config=nonccl --define=with_numa_support=true --config=clang_local '--per_file_copt=external/onednn/.*[.]cpp@-std=c++17' '--per_file_copt=external/jpegxl/.*enc_fast_lossless.*@-O1' --host_action_env=LD_LIBRARY_PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keras-applications/1.0.8-46572922f77acd7aaa7eac6e31be4377/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-termcolor/3.3.0-ce58411ee7b9d2dd53d75e0d8899c356/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-absl-py/2.5.0-05c999ddb9a849bbe7acfa518e394329/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/grpc/1.82.0-945336eb83feb5e3f2d5d71e106e0c6a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keras-preprocessing/1.1.2-3ede73f5837d0036bf9a586f2a717ab7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-mock/5.2.0-c8f699fa8b72c0a4a034ee616be6d287/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-portpicker/1.6.0-ed282dd153d4182c042b16712dd0e70d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-wrapt/1.14.1-a9d30184c425384c2f378793bacaf171/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tblib/3.2.2-511a589823e94d4167f1825dea331f35/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-lit/18.1.8-be994f3da9b5cd44deb9a8812565040e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-gast/0.7.0-bbc4d7d14d898d817cdfee899ad445f2/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-future/1.0.0-12d9b274c23c6207d7404c1f868a5314/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-flatbuffers/25.12.19-5adef237bdb68e32e4a68cb5917f57ad/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-astor/0.8.1-923c0dc4d3b7cb72fb71a86371efe95f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-h5py/3.13.0-d80bbc94d1a601a1dd2ad3e2e75a063d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/llvm/21.1.4-37d7e3d9353a61edb860f8dbfad90370/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/llvm/21.1.4-37d7e3d9353a61edb860f8dbfad90370/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-psutil/7.2.2-a447d9a5bc90c90c347b682e03c9fa3a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-funcsigs/1.0.2-55d0f9726efce9ba1f95410db2d17465/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/c-ares/1.15.0-0059b1cc7e0798fa1c703d9a2398138c/lib:/usr/lib/jvm/java-21/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pkgconfig/1.6.0-1a452b1f4c138e07ceddf790b6a92eb5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-protobuf/6.31.1-e7f44abf3563a34f293a77228434ac3b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/re2/2021-06-01-38cbd03a394ae1ed1c84cfadefad3d5d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry/2.4.1-fb88b362dccf19ca8eb2841ccfadbd80/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/protobuf/6.31.1-469e24c819cc571ffc01231df9d32ffa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pyparsing/3.3.2-ef0d5f04085b9bf32acfce938d575c66/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pbr/7.0.3-11aa53e334bbb6eb43339e4d694263d9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libunwind/1.8.1-master-013128a89f9e7cca86b791b279fda4b0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cachecontrol/0.14.4-a59a98d19449661fbb8edc1a02fc8edc/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cleo/2.1.0-cec6976f4ef302044eb5ccd9cf2bddb0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-virtualenv/21.6.0-756561e77f5b494ef3fe250adab08a8d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-requests-toolbelt/1.0.0-bc744615927425cd14e36f2369468492/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keyring/25.7.0-9d7a1604c52a8b24197e9e57a4ccb545/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jsonschema/4.26.0-a025210fb34ec265f293559a1a37f01a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-dulwich/1.2.10-7c720e73f49587dbe4003b87891621db/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pybind11/3.0.4-9f50732a0ffa4f98fb57015a2b00b8ca/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-findpython/0.8.0-807cc65bcbcb34971c5c699cba1c4e87/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pbs-installer/2026.6.10-506456f3fff2e93e1b82a1a99e42af90/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry-plugin-export/1.10.0-88afc452ce5651486823c15dfe27f714/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pexpect/4.9.0-3f4db019c5ca9b2070575c9f70536c6e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-shellingham/1.5.4-be82153ad9c665b8e7aa768966851893/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pkginfo/1.12.1.2-f348041ef5f2407c0700773014e1d20b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-fastjsonschema/2.21.2-d98ef09235efc921f305e1c81f829bd5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cachy/0.3.0-9cef4c8b7cd4decca5652b74856707bd/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/abseil-cpp/20250814.1-1b5d8455f00af5d4a81ea02cdde96054/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-rapidfuzz/3.14.5-8b578e8aec03f2dac786b6ca3ecfaaea/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jsonschema-specifications/2025.9.1-d02baf5aa858d5cbf46a3f1ed550373b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-secretstorage/3.5.0-d6cc2aeeb1abd47bf5010b156f3b402d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-python-discovery/1.4.4-224875656cc9ad34eb21e1930a486a8e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-context/6.1.2-5b8957ab26b9b518653fff8814ab4c6b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-functools/4.5.0-26cd931c21bb4eaca537fb154a26309b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/hdf5/1.14.6-d31efede7b38841ef8b9f8a848f90a37/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-importlib-resources/7.1.0-87c86909c77d8a0052834fa7348503d1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-clikit/0.6.2-65dddd4fd75178692b7f9dc16f22f99b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tomlkit/0.15.0-d8344010107c5e2bfd8607bd2528b50c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-ptyprocess/0.7.0-08a807acbdeaf8a0210e4016950d9804/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-msgpack/1.2.1-68a5de3e0e8c9768069ca8794d85ceb9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-classes/3.4.0-d83ca159d2c773c0d731e176250dc973/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-distlib/0.4.3-e0026ae536b36d15d4015f9e440a4d88/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-backports-entry-points-selectable/1.3.0-31feb9ecebccc1a8463dd4a992b28349/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/eigen/5.0.1-5fc0214f9d987734a3d0affcb38f7444/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/pcre/8.43-7f0edb13c32a66555b25b5433dd2ceed/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-requests/2.34.2-025c1adff47bf7bb9d48f88718fdfdd9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scikit-build/0.19.1-2670f713659d6d9c9e63428a64840dad/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-referencing/0.37.0-c480bb95cc46f915d9f731eca2dae9ee/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cryptography/49.0.0-25724bd905d43a3841d4657209ea42d0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-backports-tarfile/1.2.0-179121f51cbd58ba375498d5e1ee6453/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pdm-backend/2.4.9-3bd8ae433256d64da6a788e88c2a6eaa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-singledispatch/4.1.2-f9cbf839124b741b78ab92a34fc32fe5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pastel/0.2.1-138dda9efc60d33504bff40848573209/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jeepney/0.9.0-5b392ee60b07ffb11f57303ad05804de/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-crashtest/0.4.1-70d0e1d7f2b824ccdfe6a8491f529112/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pylev/1.4.0-7eba4bedb91f262b751664788cab27e9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-chardet/7.4.3-b33984c4aa2779177349390d2c8b85d8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-rpds-py/2026.6.3-f2de2b88f77ce3e2875cf4148f4fbb3d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-attrs/26.1.0-7f33d9110f4bbe1dae6b23edf093b040/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-wheel/0.47.0-61ad5ea596cd6e7c436a9e4c1885fd0e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cffi/2.1.0-37e5970c640f05a8b20b49a73972c4f5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pysocks/1.7.1-00ca2016ed21d4af0b9feca69b86e6ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-distro/1.9.0-ac99506fd88c9183c0b0b17df4061a2d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-charset-normalizer/3.4.9-f0e42712ba08249068978ed2cfcbb2f5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scikit-build-core/1.0.1-a2ad1408729418652f335e06cb85b933/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-urllib3/2.7.0-d78f89ef235f52f601fe47d043ad088c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pycparser/3.0-e9b3062937bdaf16adf6616c4b0ab91f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-appdirs/1.4.4-57220776ff0f261705c951d8674b9763/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-filelock/3.29.7-05c5dfa441e707e01c096ed7fe580e25/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-platformdirs/4.10.0-f2e8b2543d6d54fd26870a999afae833/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry-core/2.4.1-bf09a4be4e28d8a7800f857ed9dfcc5a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-idna/3.18-db476473886141170f86642f22f4fa06/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-certifi/2026.6.17-dd79a47d7752e1c66137bef4c8f5c8a7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/giflib/5.2.1-a527ad3fd94ee0fd141f1aa1140ddd0a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools-rust/1.13.0-b337d6ca45c4dc7659c800b3f22ef057/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/pcre2/10.36-dd63a742c2ac6aacbf973ba2fec8598c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatch-fancy-pypi-readme/25.1.0-0f6e3ed10c8596b564c13b5d13d1f048/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-toml/0.10.2-3e227de483fce18774a63921a5020c12/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-semantic-version/2.10.0-5ee420943b96687ebf47f0c0ba04b189/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libjpeg-turbo/3.0.4-ec71687bc2f615d6247f8963dc6fbbcf/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rust/1.94.0-a476c8da7dc7533c0dd99cfcfbefb520/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-exceptiongroup/1.3.1-13da487028bb7455a896c6d52f358d49/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/openmpi/5.0.10-f8592f367b787e1ef2a11b2d79809a06/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libpng/1.6.44-023e6fa0640b53f2ede9af9e3b636961/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libfabric/2.1.0-b856e90ad2789c9f3ca0b428116c32dd/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/ucx/1.21.0-58b64ba4219817d492ee1e7978d8eede/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rdma-core/57.0-b845b90cbb5d46e1466ee4c4023d2acb/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gdrcopy/2.6-57edefc538efc276cb205e11e89995b4/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/xpmem/v2.6.3-20220308-26e853843d2d89608ffd6fe77c75b648/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-numpy/1.26.4-63bc1a4c80b32cb4cae7108bf4004f2b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatch-vcs/0.5.0-0dcb4bac79335ffa1258711f3e41dda1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/hwloc/2.12.2-a4de4cbfe6aa30eaa46a9523cac1d4e4/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/OpenBLAS/0.3.27-47a3c8350874a1ad9f41674752b30581/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cython/3.2.8-6a715fcf6f16ccc224d92c6fd40560b8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libpciaccess/0.16-0c1f273cbc749e81d2c6cd3781ac7ea1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/zstd/1.5.7-3d33d105dce0061a2affabfbb156d655/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/numactl/2.0.19-66dfe30a339dca923efdd4243f4033c5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatchling/1.31.0-4030ec549d1c82c2b0cef5f18fa96af0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pluggy/1.6.0-0fd75bfa4c03f78ee372363f6fd02b15/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pathspec/1.1.1-7a8ee2568b5bdae408b5e721682083ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-editables/0.6-8c8b187f379733101bcfe8acf78a3362/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-trove-classifiers/2026.6.1.19-71c879924699fdbe88616334b44e6182/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-calver/2025.10.20-b6bb5414d02c3e057e0c0f068566f2cb/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libxml2/2.9.10-e186e339339cb9854d2d7babb5add21f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/autotools/1.5-14425bfcc7b9e51ea6c54a1d0ad392a6/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/cuda/13.3.1-42e59f38775163fbe81e7a5c898846b9/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools-scm/10.2.0-18cca40f517e8ff7976812e43a765580/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-vcs-versioning/2.2.2-cd8b12bcbf8a5fdc2460579346ac29ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-typing-extensions/4.16.0-6faee870a5ccab5616ed7dcddae256f9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tomli/2.4.1-abc94ceb43778947e8a8830a6a99975a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-importlib-metadata/9.0.0-1bb6a2677ac58a148c40f5cc47d62531/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-zipp/4.1.0-e3c8e5135ec7b8cf6d7d7d079c9c22ed/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pathlib2/2.3.7.post1-1e83eab522d166fa2a27f5800d8cee12/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-more-itertools/11.1.0-75070bd54163ff0866f569652b6b80f9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-packaging/26.2-86cf170aed27d06c76ed45043d25b827/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scandir/1.10.0-3727af7576d330852ac1b0b25771bca7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-contextlib2/21.6.0-75a8d40aec06f809337eb7d37ca4b111/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-six/1.17.0-c60623093adc241e93b7acdc987cb482/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-flit-core/3.12.0-f76fdd63ae0837ca65184fffa8c808b6/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/curl/8.13.0-04f63768868ff074c0f3aac68b6dda8d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pip/26.1.2-a8f68b7ccb754f1f41f5635902962f19/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pip/26.1.2-a8f68b7ccb754f1f41f5635902962f19/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools/83.0.0-54b74ad177edab04c2d329a45044cb35/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/sqlite/3.48.0-507c76bdccf2917e64f1c155b7196897/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libffi/3.4.6-b461d13020c9e14e7e60de314b0911cc/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gdbm/1.24-8a40192222c4b0afe814f539b1117fa8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/db6/6.2.32-a42f52927cfa2d15e53d2a4cd62c6196/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libuuid/2.40-d94705b84648fd294d63bf9de23bc03b/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/xz/5.8.3-3303bcf2cc1251389e9225885f971ea3/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/expat/2.7.1-fd861297a37b8f9a1d9e1c6e2937faaa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/bz2lib/1.0.8-353e2ef1c2f1d06b28a8938defa197bc/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/zlib/1.3.2-53c061acc706fe8f28fd9477d9981c20/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rpm/4.18.0-43e4fb3f0bab5d700caac5a5f643d644/lib:/.singularity.d/libs --action_env=LD_LIBRARY_PATH=/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keras-applications/1.0.8-46572922f77acd7aaa7eac6e31be4377/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-termcolor/3.3.0-ce58411ee7b9d2dd53d75e0d8899c356/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-absl-py/2.5.0-05c999ddb9a849bbe7acfa518e394329/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/grpc/1.82.0-945336eb83feb5e3f2d5d71e106e0c6a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keras-preprocessing/1.1.2-3ede73f5837d0036bf9a586f2a717ab7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-mock/5.2.0-c8f699fa8b72c0a4a034ee616be6d287/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-portpicker/1.6.0-ed282dd153d4182c042b16712dd0e70d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-wrapt/1.14.1-a9d30184c425384c2f378793bacaf171/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tblib/3.2.2-511a589823e94d4167f1825dea331f35/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-lit/18.1.8-be994f3da9b5cd44deb9a8812565040e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-gast/0.7.0-bbc4d7d14d898d817cdfee899ad445f2/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-future/1.0.0-12d9b274c23c6207d7404c1f868a5314/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-flatbuffers/25.12.19-5adef237bdb68e32e4a68cb5917f57ad/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-astor/0.8.1-923c0dc4d3b7cb72fb71a86371efe95f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/flatbuffers/25.9.23-924cdd667c776fd73e00352b00b243ef/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-h5py/3.13.0-d80bbc94d1a601a1dd2ad3e2e75a063d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/llvm/21.1.4-37d7e3d9353a61edb860f8dbfad90370/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/llvm/21.1.4-37d7e3d9353a61edb860f8dbfad90370/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-psutil/7.2.2-a447d9a5bc90c90c347b682e03c9fa3a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-funcsigs/1.0.2-55d0f9726efce9ba1f95410db2d17465/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/c-ares/1.15.0-0059b1cc7e0798fa1c703d9a2398138c/lib:/usr/lib/jvm/java-21/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pkgconfig/1.6.0-1a452b1f4c138e07ceddf790b6a92eb5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-protobuf/6.31.1-e7f44abf3563a34f293a77228434ac3b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/re2/2021-06-01-38cbd03a394ae1ed1c84cfadefad3d5d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry/2.4.1-fb88b362dccf19ca8eb2841ccfadbd80/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/protobuf/6.31.1-469e24c819cc571ffc01231df9d32ffa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pyparsing/3.3.2-ef0d5f04085b9bf32acfce938d575c66/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pbr/7.0.3-11aa53e334bbb6eb43339e4d694263d9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libunwind/1.8.1-master-013128a89f9e7cca86b791b279fda4b0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cachecontrol/0.14.4-a59a98d19449661fbb8edc1a02fc8edc/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cleo/2.1.0-cec6976f4ef302044eb5ccd9cf2bddb0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-virtualenv/21.6.0-756561e77f5b494ef3fe250adab08a8d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-requests-toolbelt/1.0.0-bc744615927425cd14e36f2369468492/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-keyring/25.7.0-9d7a1604c52a8b24197e9e57a4ccb545/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jsonschema/4.26.0-a025210fb34ec265f293559a1a37f01a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-dulwich/1.2.10-7c720e73f49587dbe4003b87891621db/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pybind11/3.0.4-9f50732a0ffa4f98fb57015a2b00b8ca/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-findpython/0.8.0-807cc65bcbcb34971c5c699cba1c4e87/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pbs-installer/2026.6.10-506456f3fff2e93e1b82a1a99e42af90/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry-plugin-export/1.10.0-88afc452ce5651486823c15dfe27f714/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pexpect/4.9.0-3f4db019c5ca9b2070575c9f70536c6e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-shellingham/1.5.4-be82153ad9c665b8e7aa768966851893/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pkginfo/1.12.1.2-f348041ef5f2407c0700773014e1d20b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-fastjsonschema/2.21.2-d98ef09235efc921f305e1c81f829bd5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cachy/0.3.0-9cef4c8b7cd4decca5652b74856707bd/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/abseil-cpp/20250814.1-1b5d8455f00af5d4a81ea02cdde96054/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-rapidfuzz/3.14.5-8b578e8aec03f2dac786b6ca3ecfaaea/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jsonschema-specifications/2025.9.1-d02baf5aa858d5cbf46a3f1ed550373b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-secretstorage/3.5.0-d6cc2aeeb1abd47bf5010b156f3b402d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-python-discovery/1.4.4-224875656cc9ad34eb21e1930a486a8e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-context/6.1.2-5b8957ab26b9b518653fff8814ab4c6b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-functools/4.5.0-26cd931c21bb4eaca537fb154a26309b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/hdf5/1.14.6-d31efede7b38841ef8b9f8a848f90a37/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-importlib-resources/7.1.0-87c86909c77d8a0052834fa7348503d1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-clikit/0.6.2-65dddd4fd75178692b7f9dc16f22f99b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tomlkit/0.15.0-d8344010107c5e2bfd8607bd2528b50c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-ptyprocess/0.7.0-08a807acbdeaf8a0210e4016950d9804/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-msgpack/1.2.1-68a5de3e0e8c9768069ca8794d85ceb9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jaraco-classes/3.4.0-d83ca159d2c773c0d731e176250dc973/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-distlib/0.4.3-e0026ae536b36d15d4015f9e440a4d88/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-backports-entry-points-selectable/1.3.0-31feb9ecebccc1a8463dd4a992b28349/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/eigen/5.0.1-5fc0214f9d987734a3d0affcb38f7444/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/pcre/8.43-7f0edb13c32a66555b25b5433dd2ceed/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-requests/2.34.2-025c1adff47bf7bb9d48f88718fdfdd9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scikit-build/0.19.1-2670f713659d6d9c9e63428a64840dad/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-referencing/0.37.0-c480bb95cc46f915d9f731eca2dae9ee/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cryptography/49.0.0-25724bd905d43a3841d4657209ea42d0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-backports-tarfile/1.2.0-179121f51cbd58ba375498d5e1ee6453/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pdm-backend/2.4.9-3bd8ae433256d64da6a788e88c2a6eaa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-singledispatch/4.1.2-f9cbf839124b741b78ab92a34fc32fe5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pastel/0.2.1-138dda9efc60d33504bff40848573209/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-jeepney/0.9.0-5b392ee60b07ffb11f57303ad05804de/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-crashtest/0.4.1-70d0e1d7f2b824ccdfe6a8491f529112/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pylev/1.4.0-7eba4bedb91f262b751664788cab27e9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-chardet/7.4.3-b33984c4aa2779177349390d2c8b85d8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-rpds-py/2026.6.3-f2de2b88f77ce3e2875cf4148f4fbb3d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-attrs/26.1.0-7f33d9110f4bbe1dae6b23edf093b040/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-wheel/0.47.0-61ad5ea596cd6e7c436a9e4c1885fd0e/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cffi/2.1.0-37e5970c640f05a8b20b49a73972c4f5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pysocks/1.7.1-00ca2016ed21d4af0b9feca69b86e6ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-distro/1.9.0-ac99506fd88c9183c0b0b17df4061a2d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-charset-normalizer/3.4.9-f0e42712ba08249068978ed2cfcbb2f5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scikit-build-core/1.0.1-a2ad1408729418652f335e06cb85b933/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-urllib3/2.7.0-d78f89ef235f52f601fe47d043ad088c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pycparser/3.0-e9b3062937bdaf16adf6616c4b0ab91f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-appdirs/1.4.4-57220776ff0f261705c951d8674b9763/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-filelock/3.29.7-05c5dfa441e707e01c096ed7fe580e25/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-platformdirs/4.10.0-f2e8b2543d6d54fd26870a999afae833/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-poetry-core/2.4.1-bf09a4be4e28d8a7800f857ed9dfcc5a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-idna/3.18-db476473886141170f86642f22f4fa06/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-certifi/2026.6.17-dd79a47d7752e1c66137bef4c8f5c8a7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/giflib/5.2.1-a527ad3fd94ee0fd141f1aa1140ddd0a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools-rust/1.13.0-b337d6ca45c4dc7659c800b3f22ef057/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/pcre2/10.36-dd63a742c2ac6aacbf973ba2fec8598c/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatch-fancy-pypi-readme/25.1.0-0f6e3ed10c8596b564c13b5d13d1f048/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-toml/0.10.2-3e227de483fce18774a63921a5020c12/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-semantic-version/2.10.0-5ee420943b96687ebf47f0c0ba04b189/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libjpeg-turbo/3.0.4-ec71687bc2f615d6247f8963dc6fbbcf/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rust/1.94.0-a476c8da7dc7533c0dd99cfcfbefb520/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-exceptiongroup/1.3.1-13da487028bb7455a896c6d52f358d49/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/openmpi/5.0.10-f8592f367b787e1ef2a11b2d79809a06/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libpng/1.6.44-023e6fa0640b53f2ede9af9e3b636961/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libfabric/2.1.0-b856e90ad2789c9f3ca0b428116c32dd/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/ucx/1.21.0-58b64ba4219817d492ee1e7978d8eede/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rdma-core/57.0-b845b90cbb5d46e1466ee4c4023d2acb/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gdrcopy/2.6-57edefc538efc276cb205e11e89995b4/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/xpmem/v2.6.3-20220308-26e853843d2d89608ffd6fe77c75b648/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-numpy/1.26.4-63bc1a4c80b32cb4cae7108bf4004f2b/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatch-vcs/0.5.0-0dcb4bac79335ffa1258711f3e41dda1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/hwloc/2.12.2-a4de4cbfe6aa30eaa46a9523cac1d4e4/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/OpenBLAS/0.3.27-47a3c8350874a1ad9f41674752b30581/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-cython/3.2.8-6a715fcf6f16ccc224d92c6fd40560b8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libpciaccess/0.16-0c1f273cbc749e81d2c6cd3781ac7ea1/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/zstd/1.5.7-3d33d105dce0061a2affabfbb156d655/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/numactl/2.0.19-66dfe30a339dca923efdd4243f4033c5/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-hatchling/1.31.0-4030ec549d1c82c2b0cef5f18fa96af0/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pluggy/1.6.0-0fd75bfa4c03f78ee372363f6fd02b15/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pathspec/1.1.1-7a8ee2568b5bdae408b5e721682083ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-editables/0.6-8c8b187f379733101bcfe8acf78a3362/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-trove-classifiers/2026.6.1.19-71c879924699fdbe88616334b44e6182/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-calver/2025.10.20-b6bb5414d02c3e057e0c0f068566f2cb/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libxml2/2.9.10-e186e339339cb9854d2d7babb5add21f/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/autotools/1.5-14425bfcc7b9e51ea6c54a1d0ad392a6/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/cuda/13.3.1-42e59f38775163fbe81e7a5c898846b9/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools-scm/10.2.0-18cca40f517e8ff7976812e43a765580/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-vcs-versioning/2.2.2-cd8b12bcbf8a5fdc2460579346ac29ec/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-typing-extensions/4.16.0-6faee870a5ccab5616ed7dcddae256f9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-tomli/2.4.1-abc94ceb43778947e8a8830a6a99975a/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-importlib-metadata/9.0.0-1bb6a2677ac58a148c40f5cc47d62531/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-zipp/4.1.0-e3c8e5135ec7b8cf6d7d7d079c9c22ed/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pathlib2/2.3.7.post1-1e83eab522d166fa2a27f5800d8cee12/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-more-itertools/11.1.0-75070bd54163ff0866f569652b6b80f9/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-packaging/26.2-86cf170aed27d06c76ed45043d25b827/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-scandir/1.10.0-3727af7576d330852ac1b0b25771bca7/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-contextlib2/21.6.0-75a8d40aec06f809337eb7d37ca4b111/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-six/1.17.0-c60623093adc241e93b7acdc987cb482/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-flit-core/3.12.0-f76fdd63ae0837ca65184fffa8c808b6/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/curl/8.13.0-04f63768868ff074c0f3aac68b6dda8d/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pip/26.1.2-a8f68b7ccb754f1f41f5635902962f19/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-pip/26.1.2-a8f68b7ccb754f1f41f5635902962f19/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/py3-setuptools/83.0.0-54b74ad177edab04c2d329a45044cb35/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/python3/3.12.13-94ad91f6663a74a1d7116c37a04d3863/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/sqlite/3.48.0-507c76bdccf2917e64f1c155b7196897/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libffi/3.4.6-b461d13020c9e14e7e60de314b0911cc/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gdbm/1.24-8a40192222c4b0afe814f539b1117fa8/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/db6/6.2.32-a42f52927cfa2d15e53d2a4cd62c6196/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/libuuid/2.40-d94705b84648fd294d63bf9de23bc03b/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/xz/5.8.3-3303bcf2cc1251389e9225885f971ea3/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/expat/2.7.1-fd861297a37b8f9a1d9e1c6e2937faaa/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/bz2lib/1.0.8-353e2ef1c2f1d06b28a8938defa197bc/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/zlib/1.3.2-53c061acc706fe8f28fd9477d9981c20/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/lib64:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/lib:/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/rpm/4.18.0-43e4fb3f0bab5d700caac5a5f643d644/lib:/.singularity.d/libs --define=zendnn=true --define=build_with_mkl_dnn_only=false --define=build_with_mkl=false --define=enable_mkl=false --define=tensorflow_mkldnn_contraction_kernel=0 --define=build_with_mkl_opensource=false //third_party/py/numpy
$TEST_TMPDIR defined: output root default is '/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/tensorflow-sources/2.21.0-3005ac050ba8566774930969b2d5f9cb/build' and max_idle_secs default is '15'.
Extracting Bazel installation...
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
ERROR: While parsing option --jobs=-j: Parameter '-j' does not follow correct syntax. This flag takes an integer, or a keyword ("auto", "HOST_CPUS", "HOST_RAM"), optionally followed by an operation ([-|*]) eg. "auto", "HOST_CPUS*.5".
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.rDu0nn (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.rDu0nn (%build)

Building py3-torch-cuda(build). Log can be found in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/py3-torch-cuda/2.13.0-b3431d24c733594755770abd2aeb1ed4/log.


@cmsbuild

Copy link
Copy Markdown
Contributor

Pull request #10724 was updated.

@akritkbehera

Copy link
Copy Markdown
Contributor Author

please test with cms-sw/cmssw#51507 for el9_aarch64_gcc13

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8b927c/54913/summary.html
COMMIT: d0b749c
CMSSW: CMSSW_20_1_X_2026-07-24-2300/el9_aarch64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10724/54913/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

+ perl -p -i -e 's|DQMServices/Core/interface/||' ROOTFilePB.pb.cc fastHadd.cc
++ root-config --cflags --libs
+ g++ -O2 -o /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/162021b900d432bb72f8cde54cf0fb4f/opt/cmssw/el9_aarch64_gcc13/external/fasthadd/2.4-162021b900d432bb72f8cde54cf0fb4f/bin/fastHadd ROOTFilePB.pb.cc ./fastHadd.cc -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/protobuf/6.31.1-469e24c819cc571ffc01231df9d32ffa/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/abseil-cpp/20250814.1-1b5d8455f00af5d4a81ea02cdde96054/include -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/protobuf/6.31.1-469e24c819cc571ffc01231df9d32ffa/lib -lprotobuf -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/abseil-cpp/20250814.1-1b5d8455f00af5d4a81ea02cdde96054/lib -pthread -std=c++20 -fsigned-char -I/cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02951/el9_aarch64_gcc13/lcg/root/6.36.13-0f6910381655efa33dd9a915c6f173c9/include -L/cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02951/el9_aarch64_gcc13/lcg/root/6.36.13-0f6910381655efa33dd9a915c6f173c9/lib -lCore -lImt -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lROOTVecOps -lTree -lTreePlayer -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lROOTNTuple -lMultiProc -lROOTDataFrame -lROOTNTupleUtil -Wl,-rpath,/cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02951/el9_aarch64_gcc13/lcg/root/6.36.13-0f6910381655efa33dd9a915c6f173c9/lib -pthread -lm -ldl -rdynamic
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/bin/../lib/gcc/aarch64-redhat-linux-gnu/13.4.0/../../../../aarch64-redhat-linux-gnu/bin/ld: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_aarch64_gcc13/external/fasthadd/2.4-162021b900d432bb72f8cde54cf0fb4f/cmsdist-tmp/ccOvEWzJ.o: undefined reference to symbol '_ZN4absl12lts_2025081412log_internal17MakeCheckOpStringIPKvS4_EEPKcT_T0_S6_'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/gcc/13.4.0-6908cfdf803923e783448096ca4f0923/bin/../lib/gcc/aarch64-redhat-linux-gnu/13.4.0/../../../../aarch64-redhat-linux-gnu/bin/ld: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_aarch64_gcc13/external/abseil-cpp/20250814.1-1b5d8455f00af5d4a81ea02cdde96054/lib/libabsl_log_internal_check_op.so.2508.0.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.pOju9X (%build)

RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.pOju9X (%build)



Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update TensorFlow ?

4 participants