Building traccc against non-installed ROCm version

[StewMH]

With main I have now managed to build some of ACTS + traccc_alpaka_hip against ROCm 7.13.0 + clang 22 on a machine with an AMD GPU + ROCm installed in the standard /opt/rocm
Recipe:

asetup main,latest,Athena,clang
source /cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/setup.sh
export HIPCXX=`which amdclang++`
cmake ../acts/Traccc \
-DCMAKE_BUILD_TYPE=Release --preset alpaka-fp32 \
-Dalpaka_ACC_GPU_HIP_ENABLE=ON \
-Dalpaka_DISABLE_VENDOR_RNG=ON \
-DTRACCC_USE_ROOT=OFF \
-DTRACCC_SETUP_ROCTHRUST=ON \
-DTRACCC_USE_SYSTEM_ROCTHRUST=ON \
-DTRACCC_BUILD_HIP=ON \
-DCMAKE_HIP_COMPILE_FEATURES=hip_std_20 \
-DDETRAY_BUILD_VALIDATION_TOOLS=ON     \
-DACTS_ENABLE_CUDA=OFF \
-DTRACCC_SETUP_GOOGLETEST=ON \
-DTRACCC_FAIL_ON_WARNINGS=OFF \
-DTRACCC_SUPPORTED_DETECTORS="default_detector;odd_detector;telescope_detector;toy_detector;itk_detector" \
-DDETRAY_GENERATE_METADATA="itk_metadata" \
-DCMAKE_HIP_FLAGS="--gcc-toolchain=`which g++ | sed 's|/bin/g++||'`" \
-DCMAKE_PREFIX_PATH=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/cmake/ \
-DHIPToolkit_INCLUDE_DIR=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/include/ \
-DHIPToolkit_amdhip64_LIBRARY=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/libamdhip64.so \
-DCMAKE_HIP_FLAGS="--gcc-toolchain=`which g++ | sed 's|/bin/g++||'`" \
-DAMDDeviceLibs_DIR=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/cmake/ \
-Damd_comgr_DIR=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/cmake/amd_comgr/ \
-Dhip_DIR=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/cmake/hip \
-Dhsa-runtime64_DIR=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9/lib/cmake/hsa-runtime64

It's very annoying that we have to set CMAKE_PREFIX_PATH and 8 additional paths. Unclear to me where this is coming from: the assumption that rocm is in /opt/rocm is very strong.

After this step, it fails with an internal compiler error:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/clang-23 -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -O3 -Werror=atomic-alignment -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name combinatorial_kalman_filter_algorithm.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -fno-threadsafe-statics -mllvm -amdgpu-internalize-symbols -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/amdgcn/bitcode/oclc_abi_version_600.bc -target-cpu gfx90a -debugger-tuning=gdb -fdebug-compilation-dir=/scratch/smh/cvmfs_rocm/build_acts/bld/device/alpaka -fcoverage-compilation-dir=/scratch/smh/cvmfs_rocm/build_acts/bld/device/alpaka -resource-dir /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/lib/clang/23 -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/lib/clang/23/include/cuda_wrappers -idirafter /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/include -include __clang_hip_runtime_wrapper.h -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/vecmem-build/core/CMakeFiles -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/vecmem-src/core/include -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/covfie-src/lib/core -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/nlohmann_json-src/include -isystem /usr/include/boost1.78 -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/eigen-src -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/alpaka-src/include -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/vecmem-build/hip/CMakeFiles -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/vecmem-src/hip/include -isystem /scratch/smh/cvmfs_rocm/build_acts/bld/_deps/covfie-src/lib/hip -D ALPAKA_ACC_CPU_B_SEQ_T_THREADS_ENABLED -D ALPAKA_ACC_GPU_HIP_ENABLED -D ALPAKA_BLOCK_SHARED_DYN_MEMBER_ALLOC_KIB=47 -D ALPAKA_DEBUG=0 -D ALPAKA_DISABLE_VENDOR_RNG -D ALPAKA_HAS_STD_ATOMIC_REF -D DETRAY_ALGEBRA_ARRAY -D DETRAY_CUSTOM_SCALARTYPE=float -D DETRAY_INTERNAL_USE_NO_UNIQUE_ADDRESS_ANNOTATION -D DETRAY_LOG_LVL=-1 -D THRUST_IGNORE_CUB_VERSION_CHECK -D USE_PROF_API=1 -D VECMEM_DEBUG_MSG_LVL=0 -D VECMEM_HAVE_PMR_MEMORY_RESOURCE -D VECMEM_SOURCE_DIR_LENGTH=47 -D VECMEM_SUPPORT_POSIX_ATOMIC_REF -D __HIP_PLATFORM_AMD__ -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__ -D __HIP_ROCclr__=1 -D traccc_alpaka_EXPORTS -I /scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/device/alpaka/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/core/include -I /scratch/smh/cvmfs_rocm/build_acts/bld/core/CMakeFiles/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Detray/core/include -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/core/CMakeFiles -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/core/codegen/50a6c1ef5aadcbd48eb38bfd0e81e3cafd5df725 -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/core/codegen/a2f005b083cf8faabaaa5f19f911494d78be0d8a -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/core/codegen/d3173c582c4b9fb20fc600956089329fb53846db -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/core/codegen/48c37d545b823e72db3942c03873b2cf62cc4aa8 -I /scratch/smh/cvmfs_rocm/build_acts/acts/Detray/plugins/algebra/array/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Detray/detectors/include -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Detray/detectors/CMakeFiles/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Detray/io/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Core/include -I /scratch/smh/cvmfs_rocm/build_acts/bld/extern/acts/acts/Core/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/device/common/include -I /scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/device/hip_utils/include -D NDEBUG -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0 -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/x86_64-pc-linux-gnu -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/backward -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0 -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/x86_64-pc-linux-gnu -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../include/c++/15.2.0/backward -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/lib/clang/23/include -internal-isystem /usr/local/include -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../x86_64-pc-linux-gnu/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/lib/clang/23/include -internal-isystem /usr/local/include -internal-isystem /cvmfs/sft.cern.ch/lcg/releases/gcc/15.2.0-35657/x86_64-el9/lib/gcc/x86_64-pc-linux-gnu/15.2.0/../../../../x86_64-pc-linux-gnu/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wall -Wextra -Wshadow -Wunused-local-typedefs -pedantic -std=c++20 -fdeprecated-macro -fno-autolink -ferror-limit 19 -fmessage-length=204 -fhip-new-launch-api -fgnuc-version=4.2.1 -fno-implicit-modules -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -vectorize-loops -vectorize-slp -cuid=95a6990421880fa6 -fcuda-allow-variadic-functions -faddrsig -fdwarf2-cfi-asm -o /tmp/combinatorial_kalman_filter_algorithm-gfx90a-626fdf.o -x hip /scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/device/alpaka/src/finding/combinatorial_kalman_filter_algorithm.cpp
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'CallGraph Pass Manager' on module '/scratch/smh/cvmfs_rocm/build_acts/acts/Traccc/device/alpaka/src/finding/combinatorial_kalman_filter_algorithm.cpp'.
4.	Running pass 'Greedy Register Allocator' on function '@_ZN6alpaka6detail9gpuKernelIN6traccc6alpaka7kernels25progressive_kalman_filterIN6detray10propagatorINS6_10rk_stepperIN6covfie10field_viewINS9_7backend8constantINS9_6vector8vector_dIfLm3EEESF_EEEENS6_5arrayIfEENS6_16constrained_stepIfEENS6_17stepper_rk_policyIfEENS6_8stepping14void_inspectorELj1EEENS6_17caching_navigatorIKNS6_8detectorINS6_12itk_metadataISJ_EENS2_7details31device_detector_container_typesEEELm8ENS6_10navigation14void_inspectorENS6_14intersection2DINS6_18surface_descriptorINS6_6detail11typed_indexINSU_7mask_idENS13_11index_rangeIjLb1EjLj268435200ELj255EEEjLj4026531840ELj268435455EEENS14_INSU_11material_idEjjLj4026531840ELj268435455EEEjtEESJ_Lb0EEEEENS6_11actor_chainIJNS6_5actor17pathlimit_aborterIfEENS6_15composite_actorINS1F_21parameter_transporterISJ_EEJNS1F_29pointwise_material_interactorISJ_EENS1F_16parameter_setterISJ_EEEEENS1F_16momentum_aborterIfEENS2_11ckf_aborterEEEEEEEENS_22AccGpuUniformCudaHipRtINS_8ApiHipRtESt17integral_constantImLm1EEjEEJNS2_14finding_configEPNS6_11dmulti_viewIJN6vecmem4data11vector_viewIKNS6_17volume_descriptorINSU_11geo_objectsENS14_INSU_8accel_idEjjLj4026531840ELj268435455EEES1A_EEEENS25_IKNS6_11source_linkIS1B_EEEENS25_IKNS6_7algebra7generic4math10transform3ImfNS2H_5array11matrix_typeENS2L_12storage_typeEEEEENS22_IJNS25_IKNS6_4maskINS6_9annulus2DESJ_tEEEENS25_IKNS2R_INS6_11rectangle2DESJ_tEEEENS25_IKNS2R_INS6_21concentric_cylinder2DESJ_tEEEENS25_IKNS2R_INS6_6ring2DESJ_tEEEEEEENS22_IJNS22_IJNS25_IKjEENS25_IKNS6_4bins6singleINS6_13material_slabIfEEEEEENS25_IKNS16_IjLb1EmLm18446744069414584320ELm4294967295EEEEENS25_IKfEEEEES3N_EEENS22_IJNS22_IJS3A_NS25_IKS1B_EEEEENS22_IJNS22_IJS3A_NS22_IJNS25_IKNS3B_13dynamic_arrayIS1B_E4dataEEES3Q_EEES3K_S3M_EEENS25_IKNS2R_IS30_SJ_hEEEEEEES42_NS22_IJNS22_IJS3A_NS25_IKNS3C_IjEEEES3K_S3M_EEES41_EEEEEEEEESH_NS24_18jagged_vector_viewIS1B_EENS2_6device33progressive_kalman_filter_payloadEEEEvNS_3VecINS_5trait7DimTypeIT0_vE4typeENS4G_7IdxTypeIS4I_vE4typeEEET_DpT1_'
 #0 0x00007fd4f8d1341b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x91341b)
 #1 0x00007fd4f8d1017f SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #2 0x00007fd4f8d10238 SignalHandlerTerminate(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007fd4f7c3fb30 __restore_rt (/lib64/libc.so.6+0x3fb30)
 #4 0x00007fd4f95a9dac llvm::SplitEditor::extendPHIKillRanges() (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x11a9dac)
 #5 0x00007fd4f95b2435 llvm::SplitEditor::finish(llvm::SmallVectorImpl<unsigned int>*) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x11b2435)
 #6 0x00007fd4f94ed9cf llvm::RAGreedy::tryBlockSplit(llvm::LiveInterval const&, llvm::AllocationOrder&, llvm::SmallVectorImpl<llvm::Register>&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x10ed9cf)
 #7 0x00007fd4f94f6232 llvm::RAGreedy::trySplit(llvm::LiveInterval const&, llvm::AllocationOrder&, llvm::SmallVectorImpl<llvm::Register>&, llvm::SmallSet<llvm::Register, 16u, std::less<llvm::Register>> const&) (.part.0) RegAllocGreedy.cpp:0:0
 #8 0x00007fd4f94fa97a llvm::RAGreedy::selectOrSplitImpl(llvm::LiveInterval const&, llvm::SmallVectorImpl<llvm::Register>&, llvm::SmallSet<llvm::Register, 16u, std::less<llvm::Register>>&, llvm::SmallVector<std::pair<llvm::LiveInterval const*, llvm::MCRegister>, 8u>&, unsigned int) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x10fa97a)
 #9 0x00007fd4f94faf51 llvm::RAGreedy::selectOrSplit(llvm::LiveInterval const&, llvm::SmallVectorImpl<llvm::Register>&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x10faf51)
#10 0x00007fd4f94d2201 llvm::RegAllocBase::allocatePhysRegs() (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x10d2201)
#11 0x00007fd4f94f6ef1 llvm::RAGreedy::run(llvm::MachineFunction&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0x10f6ef1)
#12 0x00007fd4f94f80cb (anonymous namespace)::RAGreedyLegacy::runOnMachineFunction(llvm::MachineFunction&) RegAllocGreedy.cpp:0:0
#13 0x00007fd4f92fa950 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0xefa950)
#14 0x00007fd4f8ef1659 llvm::FPPassManager::runOnFunction(llvm::Function&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0xaf1659)
#15 0x00007fd4fad52ed1 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#16 0x00007fd4f8ef0bee llvm::legacy::PassManagerImpl::run(llvm::Module&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libLLVM.so.23.0git+0xaf0bee)
#17 0x00007fd5016d58a6 clang::emitBackendOutput(clang::CompilerInstance&, clang::CodeGenOptions&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>, clang::BackendConsumer*) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0x22d58a6)
#18 0x00007fd501aca3b1 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0x26ca3b1)
#19 0x00007fd4fff9ca7c clang::ParseAST(clang::Sema&, bool, bool) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0xb9ca7c)
#20 0x00007fd5024fd28a clang::FrontendAction::Execute() (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0x30fd28a)
#21 0x00007fd502479cf3 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0x3079cf3)
#22 0x00007fd5025a8bea clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/../lib/libclang-cpp.so.23.0git+0x31a8bea)
#23 0x000000000041396d cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/clang-23+0x41396d)
#24 0x000000000040b1ca ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) driver.cpp:0:0
#25 0x000000000040f390 clang_main(int, char**, llvm::ToolContext const&) (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/clang-23+0x40f390)
#26 0x000000000040abe9 main (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/clang-23+0x40abe9)
#27 0x00007fd4f7c2a610 __libc_start_call_main (/lib64/libc.so.6+0x2a610)
#28 0x00007fd4f7c2a6c0 __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a6c0)
#29 0x000000000040ac2e _start (/cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin/clang-23+0x40ac2e)
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: clang frontend command failed with exit code -2 (use -v to see invocation)
AMD clang version 23.0.0git (https://github.com/ROCm/llvm-project.git 43215c73116c407735c85a180d174f718798c328+PATCHED:2506c552d8428e2cc1778bef048b20f818e06bb3)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/sft.cern.ch/lcg/releases/rocm/7.13.0-6ad09/x86_64-el9-gcc15-opt/lib/llvm/bin
clang++: note: diagnostic msg: Error generating preprocessed source(s).
make[2]: *** [device/alpaka/CMakeFiles/traccc_alpaka.dir/build.make:234: device/alpaka/CMakeFiles/traccc_alpaka.dir/src/finding/combinatorial_kalman_filter_algorithm.cpp.o] Error 254

I don't think we need to worry too much about this yet, until we see something similar in the CI.

[paulgessinger]

I seem to have managed to compile ok against a rocm installed and relocated by Spack. That's my vibe coded package that takes the binaries and relocated them i believe. Maybe there's something it's doing that enables this?

[StewMH]

But do you have two local rocms? I think that's the initial issue here (and then I'm straight into the ICE). I'm not familiar with Spack but in the short-term for ATLAS integration I think we need to do everything through plain cmake.

[paulgessinger]

Spack only communicates with our build through CMake discovery. The acts/traccc build is unaware of it.

You're right of course that Spack is out of scope of ATLAS.

What I meant to say is that I was able to build without (as far as I can tell) any additional variables being set or special treatment.

It's possible I'm missing something though.

[krasznaa]

I have managed to build code with CVMFS's ROCm installation. It's a fair amount of pfaff at the moment, but it is (sort of) possible.

The following is not quite the ideal way for working in an Athena environment (for now I've been doing my developments independent of Athena), but is sort of close (traccc should not build TBB for instance in this setup):

asetup Athena,main,latest,gcc15
export ROCM_PATH=/cvmfs/sft.cern.ch/lcg/contrib/rocm/7.13.0/x86_64-el9
source $ROCM_PATH/setup.sh
unset HIPCXX
source /cvmfs/sft.cern.ch/lcg/views/LCG_110_ATLAS_4/x86_64-el9-gcc15-opt/setup.sh
cmake --preset gpu-devel -S acts -B build
cmake --build build/

The preset in question is this one:

{
   "version": 4,
   "configurePresets": [
      {
         "name": "gpu-devel",
         "displayName": "pcadp04 GPU Development",
         "warnings": {
            "deprecated": true
         },
         "binaryDir": "build",
         "cacheVariables": {
            "ROCM_PATH": "$env{ROCM_PATH}",
            "ROCM_ROOT": "$env{ROCM_PATH}",
            "CMAKE_BUILD_TYPE": "RelWithDebInfo",
            "CMAKE_CXX_STANDARD": "20",
            "CMAKE_CUDA_STANDARD": "20",
            "CMAKE_CUDA_ARCHITECTURES": "native",
            "CMAKE_HIP_STANDARD": "20",
            "CMAKE_HIP_ARCHITECTURES": "native",
            "CMAKE_EXPORT_COMPILE_COMMANDS": "ON",
            "BUILD_TESTING": "ON",
            "ACTS_ENABLE_LOG_FAILURE_THRESHOLD": "OFF",
            "ACTS_BUILD_FATRAS": "OFF",
            "ACTS_BUILD_UNITTESTS": "OFF",
            "ACTS_BUILD_INTEGRATIONTESTS": "OFF",
            "ACTS_BUILD_PYTHON_BINDINGS": "ON",
            "ACTS_GENERATE_PYTHON_STUBS": "OFF",
            "ACTS_BUILD_PLUGIN_TRACCC": "ON",
            "ACTS_ENABLE_CUDA": "ON",
            "ACTS_USE_SYSTEM_EIGEN3": "OFF",
            "VECMEM_BUILD_CUDA_LIBRARY": "ON",
            "VECMEM_BUILD_HIP_LIBRARY": "ON",
            "DETRAY_BUILD_CUDA": "ON",
            "DETRAY_BUILD_BENCHMARKS": "ON",
            "DETRAY_BUILD_VALIDATION_TOOLS": "ON",
            "DETRAY_SET_LOGGING": "NONE",
            "TRACCC_SETUP_TBB": "ON",
            "TRACCC_SETUP_ROCTHRUST": "OM",
            "TRACCC_BUILD_CUDA": "ON",
            "TRACCC_BUILD_HIP": "ON",
            "TRACCC_BUILD_ALPAKA": "ON",
            "alpaka_ACC_GPU_HIP_ENABLE": "ON",
            "TRACCC_SUPPORTED_DETECTORS": "default_detector;itk_detector",
            "TRACCC_BUILD_PERFORMANCE": "ON",
            "TRACCC_BUILD_SIMULATION": "ON",
            "TRACCC_BUILD_TESTING": "ON",
            "TRACCC_BUILD_EXAMPLES": "ON",
            "TRACCC_BUILD_BENCHMARKS": "ON"
         }
      }
   ]
}

As you can see, I'm also "building Acts" in this recipe, not traccc. As I said recently, we need to sit together a few of us in the next few weeks to sort out how this sort of development should go. Both on top of Athena and in a standalone mode. 🤔

There are a lot of details hidden in the preset. But the main one is that ROCM_PATH is a pretty important environment and cache variable for many HIP-using CMake modules. (ROCM_ROOT is only used by one rocThrust file. I believe because of a mistake, that should've used ROCM_PATH as well...)

[paulgessinger]

Are you all observing build failures or runtime failures when there are issues?

[krasznaa]

I've not seen a build failure like that. I believe Stewart's build setup is not quite correct, and the local ROCm installation from /opt still interferes with the one coming from CVMFS. 🤔

The ROCm 7.13 version on CVMFS is far from perfect unfortunately. 😦 But with the correct environment, I believe we will be able to make it work for us. This is sort of an ongoing effort together with SFT. (A large part of it being us figuring out how we'd want to include ROCm in ATLAS LCG layers exactly. This has a number of aspects to it...)

[StewMH]

I did check that I didn't get any interference from /opt/rocm in my CMakeCache.txt, but with running these several times it's possible to mess up. The internal compiler error could also just be out of memory or something.

Also note that I'm using clang and Attila is using gcc as the host compiler.

[StewMH]

I also see this with GCC locally and with clang 7.2 in the CI: #5774

[StewMH]

this is the stack trace.

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/rocm-7.2.0/lib/llvm/bin/clang-22 -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -O3 -Werror=atomic-alignment -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name combinatorial_kalman_filter_algorithm.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -fno-threadsafe-statics -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-7.2.0/lib/llvm/lib/clang/22/lib/amdgcn/bitcode/oclc_abi_version_600.bc -target-cpu gfx90a -debugger-tuning=gdb -fdebug-compilation-dir=/__w/acts/acts/build/device/alpaka -fcoverage-compilation-dir=/__w/acts/acts/build/device/alpaka -resource-dir /opt/rocm-7.2.0/lib/llvm/lib/clang/22 -internal-isystem /opt/rocm-7.2.0/lib/llvm/lib/clang/22/include/cuda_wrappers -idirafter /opt/rocm-7.2.0/lib/llvm/bin/../../../include -include __clang_hip_runtime_wrapper.h -isystem /vecmem_install/include -isystem /covfie_install/include -isystem /json_install/include -isystem /__w/acts/acts/build/_deps/eigen-src -isystem /__w/acts/acts/build/_deps/alpaka-src/include -isystem /__w/acts/acts/build/_deps/rocthrust-build/thrust/include -isystem /__w/acts/acts/build/_deps/rocthrust-src/thrust/.. -isystem /__w/acts/acts/build/_deps/rocthrust-build/deps/rocprim/include -D ALPAKA_ACC_CPU_B_SEQ_T_THREADS_ENABLED -D ALPAKA_ACC_GPU_HIP_ENABLED -D ALPAKA_BLOCK_SHARED_DYN_MEMBER_ALLOC_KIB=47 -D ALPAKA_DEBUG=0 -D ALPAKA_DISABLE_VENDOR_RNG -D ALPAKA_HAS_STD_ATOMIC_REF -D DETRAY_ALGEBRA_ARRAY -D DETRAY_CUSTOM_SCALARTYPE=float -D DETRAY_INTERNAL_USE_NO_UNIQUE_ADDRESS_ANNOTATION -D DETRAY_LOG_LVL=-1 -D THRUST_IGNORE_CUB_VERSION_CHECK -D USE_PROF_API=1 -D VECMEM_DEBUG_MSG_LVL=0 -D VECMEM_HAVE_PMR_MEMORY_RESOURCE -D VECMEM_SUPPORT_POSIX_ATOMIC_REF -D __HIP_PLATFORM_AMD__ -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__ -D __HIP_ROCclr__=1 -D traccc_alpaka_EXPORTS -I /__w/acts/acts/Traccc/device/alpaka/include -I /__w/acts/acts/Traccc/core/include -I /__w/acts/acts/build/core/CMakeFiles/include -I /__w/acts/acts/Detray/core/include -I /__w/acts/acts/build/extern/acts/acts/Detray/core/CMakeFiles -I /__w/acts/acts/build/extern/acts/acts/Detray/core/codegen/50a6c1ef5aadcbd48eb38bfd0e81e3cafd5df725 -I /__w/acts/acts/build/extern/acts/acts/Detray/core/codegen/a2f005b083cf8faabaaa5f19f911494d78be0d8a -I /__w/acts/acts/build/extern/acts/acts/Detray/core/codegen/d3173c582c4b9fb20fc600956089329fb53846db -I /__w/acts/acts/build/extern/acts/acts/Detray/core/codegen/48c37d545b823e72db3942c03873b2cf62cc4aa8 -I /__w/acts/acts/Detray/plugins/algebra/array/include -I /__w/acts/acts/Detray/detectors/include -I /__w/acts/acts/build/extern/acts/acts/Detray/detectors/CMakeFiles/include -I /__w/acts/acts/Detray/io/include -I /__w/acts/acts/Core/include -I /__w/acts/acts/build/extern/acts/acts/Core/include -I /__w/acts/acts/Traccc/device/common/include -I /__w/acts/acts/Traccc/device/hip_utils/include -D NDEBUG -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/x86_64-linux-gnu/c++/13 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/x86_64-linux-gnu/c++/13 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/backward -internal-isystem /opt/rocm-7.2.0/lib/llvm/lib/clang/22/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/rocm-7.2.0/lib/llvm/lib/clang/22/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/13/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wall -Wextra -Wshadow -Wunused-local-typedefs -Werror -pedantic -std=c++20 -fdeprecated-macro -fno-autolink -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fno-implicit-modules -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -vectorize-loops -vectorize-slp -cuid=76b0c23261c0a1a2 -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/combinatorial_kalman_filter_algorithm-gfx90a-69f959.o -x hip /__w/acts/acts/Traccc/device/alpaka/src/finding/combinatorial_kalman_filter_algorithm.cpp
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'CallGraph Pass Manager' on module '/__w/acts/acts/Traccc/device/alpaka/src/finding/combinatorial_kalman_filter_algorithm.cpp'.
4.	Running pass 'Greedy Register Allocator' on function '@_ZNK6detray10rk_stepperIN6covfie10field_viewINS1_7backend6affineINS3_6linearINS3_5clampINS3_7stridedINS1_6vector8vector_dImLm3EEENS3_5arrayINS9_IfLm3EEEmEEEEEESC_EEEEEENS_5arrayIfEENS_18unconstrained_stepIfEENS_17stepper_rk_policyIfEENS_8stepping14void_inspectorELj3EE4stepEfRNSR_5stateERKNSP_6configEbPKNS_8materialIfSt5ratioILl1ELl1EEEE'
 #0 0x0000558e6a3f3240 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x2893240)
 #1 0x0000558e6a3f043d SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #2 0x00007f944e8e4330 (/lib/x86_64-linux-gnu/libc.so.6+0x45330)
 #3 0x0000558e699677f5 llvm::SplitEditor::extendPHIKillRanges() (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1e077f5)
 #4 0x0000558e699747e6 llvm::SplitEditor::finish(llvm::SmallVectorImpl<unsigned int>*) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1e147e6)
 #5 0x0000558e698d2913 llvm::RAGreedy::tryBlockSplit(llvm::LiveInterval const&, llvm::AllocationOrder&, llvm::SmallVectorImpl<llvm::Register>&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1d72913)
 #6 0x0000558e698d8307 llvm::RAGreedy::trySplit(llvm::LiveInterval const&, llvm::AllocationOrder&, llvm::SmallVectorImpl<llvm::Register>&, llvm::SmallSet<llvm::Register, 16u, std::less<llvm::Register>> const&) (.part.0) RegAllocGreedy.cpp:0:0
 #7 0x0000558e698df30f llvm::RAGreedy::selectOrSplitImpl(llvm::LiveInterval const&, llvm::SmallVectorImpl<llvm::Register>&, llvm::SmallSet<llvm::Register, 16u, std::less<llvm::Register>>&, llvm::SmallVector<std::pair<llvm::LiveInterval const*, llvm::MCRegister>, 8u>&, unsigned int) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1d7f30f)
 #8 0x0000558e698df8f8 llvm::RAGreedy::selectOrSplit(llvm::LiveInterval const&, llvm::SmallVectorImpl<llvm::Register>&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1d7f8f8)
 #9 0x0000558e69b50553 llvm::RegAllocBase::allocatePhysRegs() (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1ff0553)
#10 0x0000558e698db3a0 llvm::RAGreedy::run(llvm::MachineFunction&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1d7b3a0)
#11 0x0000558e698dc888 (anonymous namespace)::RAGreedyLegacy::runOnMachineFunction(llvm::MachineFunction&) RegAllocGreedy.cpp:0:0
#12 0x0000558e6976467d llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x1c0467d)
#13 0x0000558e69d21282 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x21c1282)
#14 0x0000558e693588aa (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#15 0x0000558e69d21e72 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x21c1e72)
#16 0x0000558e6a67f762 (anonymous namespace)::EmitAssemblyHelper::RunCodegenPipeline(clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>&, std::unique_ptr<llvm::ToolOutputFile, std::default_delete<llvm::ToolOutputFile>>&) BackendUtil.cpp:0:0
#17 0x0000558e6a680233 clang::emitBackendOutput(clang::CompilerInstance&, clang::CodeGenOptions&, llvm::StringRef, llvm::Module*, clang::BackendAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>, clang::BackendConsumer*) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x2b20233)
#18 0x0000558e6acb5fd8 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x3155fd8)
#19 0x0000558e6c676e5c clang::ParseAST(clang::Sema&, bool, bool) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x4b16e5c)
#20 0x0000558e6b008bae clang::FrontendAction::Execute() (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x34a8bae)
#21 0x0000558e6af7818b clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x341818b)
#22 0x0000558e6b0ecce4 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0x358cce4)
#23 0x0000558e6880b735 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0xcab735)
#24 0x0000558e6880240b ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) driver.cpp:0:0
#25 0x0000558e6880562a clang_main(int, char**, llvm::ToolContext const&) (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0xca562a)
#26 0x0000558e68708aab main (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0xba8aab)
#27 0x00007f944e8c91ca (/lib/x86_64-linux-gnu/libc.so.6+0x2a1ca)
#28 0x00007f944e8c928b __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28b)
#29 0x0000558e68801e85 _start (/opt/rocm-7.2.0/lib/llvm/bin/clang-22+0xca1e85)
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
AMD clang version 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-7.2.0 26014 7b800a19466229b8479a78de19143dc33c3ab9b5)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-7.2.0/lib/llvm/bin
Configuration file: /opt/rocm-7.2.0/lib/llvm/bin/clang++.cfg
clang++: note: diagnostic msg: Error generating preprocessed source(s).

[StewMH]

Looks like this is from the O3 pass which explains why Attila doesn't see it (RelWithDebInfo is O2 -g). By dropping to O2 directly and rerunning the compilation command I can get past it. Formally I can drop the optimisation level down two ways:

1. move to RelWithDebInfo
2. add an explicit CXXFLAGS set in the ALPAKA_HIP CI target