Error running containerized DLMUSE

[dhivsrin]

Hi,

I am getting below error while running containerized version of NiChart_DLMUSE on CapeTown Site from PHENOM. The initial input image/header seems fine to me.

Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Renaming dic is saved to /cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/renamed_image/renaming.json
Loading the model...
There are 327 cases in the source folder
I am process 0 out of 1 (max process ID is 0, we start counting with 0!)
There are 327 cases that I would like to predict
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Traceback (most recent call last):
File "/opt/conda/bin/DLICV", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/DLICV/main.py", line 329, in main
predictor.predict_from_files(
File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 258, in predict_from_files
return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 351, in predict_from_data_iterator
for preprocessed in data_iterator:
File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/data_iterators.py", line 117, in preprocessing_iterator_fromfiles
[i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)]
File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/data_iterators.py", line 117, in
[i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)]
RuntimeError: CUDA error: CUDA driver version is insufficient for CUDA runtime version
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Device-side assertions were explicitly omitted for this error check; the error probably arose while initializing the DSA handlers.

Arguments:
Namespace(in_dir='/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025//CapeTown/Protocols/In', out_dir='/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025//CapeTown/Protocols/DLMUSE', device='cuda', cores='1', bids=False, clear_cache=False, dlmuse_args='', dlicv_args='', refaced_data=False, help=False)

Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/nibabel/loadsave.py", line 101, in load
stat_result = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/S4132_20110209_T1_DLICV.nii.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/bin/NiChart_DLMUSE", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/main.py", line 250, in main
run_pipeline(
File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/dlmuse_pipeline.py", line 148, in run_pipeline
apply_mask_img(df_img, in_dir, in_suff, mask_dir, mask_suff, out_dir, out_suff)
File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/MaskImage.py", line 187, in apply_mask_img
mask_img(in_img, in_mask, out_img)
File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/MaskImage.py", line 83, in mask_img
nii_mask = nib.load(mask_img)
File "/opt/conda/lib/python3.10/site-packages/nibabel/loadsave.py", line 103, in load
raise FileNotFoundError(f"No such file or no access: '{filename}'")
FileNotFoundError: No such file or no access: '/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/S4132_20110209_T1_DLICV.nii.gz

Here is my code:

singularity run --cleanenv
-B /cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025:/pipelinedir/
nichart_dlmuse-1.0.10.sif
-i ${in}
-o ${out}
-d cuda
-c 1

Please let me know if you need more details.

[euroso97]

Hi Dhivya,

Thanks for sharing the issue. Can you also share the slurm submission configuration details with us? (all the --sbatch arguments you've submitted)

[AlexanderGetka-cbica]

Hi Dhivya, I believe the FileNotFoundError is arising from the upstream error regarding CUDA driver/runtime versions. I can build the same container with support for whatever version of CUDA that CUBIC supports. Will get back to you with this soon Thanks, Alex

…

________________________________ From: Dhivya Srinivasan ***@***.***> Sent: Friday, October 3, 2025 3:09 PM To: CBICA/NiChart_DLMUSE ***@***.***> Cc: Subscribed ***@***.***> Subject: [External] [CBICA/NiChart_DLMUSE] Error running containerized DLMUSE (Issue #82) [https://avatars.githubusercontent.com/u/25521662?s=20&v=4]dhivsrin created an issue (CBICA/NiChart_DLMUSE#82)<#82> Hi, I am getting below error while running containerized version of NiChart_DLMUSE on CapeTown Site from PHENOM. The initial input image/header seems fine to me. Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. Renaming dic is saved to /cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/renamed_image/renaming.json Loading the model... There are 327 cases in the source folder I am process 0 out of 1 (max process ID is 0, we start counting with 0!) There are 327 cases that I would like to predict Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library. Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it. Traceback (most recent call last): File "/opt/conda/bin/DLICV", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/DLICV/main.py", line 329, in main predictor.predict_from_files( File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 258, in predict_from_files return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export) File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/predict_from_raw_data.py", line 351, in predict_from_data_iterator for preprocessed in data_iterator: File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/data_iterators.py", line 117, in preprocessing_iterator_fromfiles [i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)] File "/opt/conda/lib/python3.10/site-packages/nnunetv2/inference/data_iterators.py", line 117, in [i.pin_memory() for i in item.values() if isinstance(i, torch.Tensor)] RuntimeError: CUDA error: CUDA driver version is insufficient for CUDA runtime version CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Device-side assertions were explicitly omitted for this error check; the error probably arose while initializing the DSA handlers. Arguments: Namespace(in_dir='/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025//CapeTown/Protocols/In', out_dir='/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025//CapeTown/Protocols/DLMUSE', device='cuda', cores='1', bids=False, clear_cache=False, dlmuse_args='', dlicv_args='', refaced_data=False, help=False) Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/nibabel/loadsave.py", line 101, in load stat_result = os.stat(filename) FileNotFoundError: [Errno 2] No such file or directory: '/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/S4132_20110209_T1_DLICV.nii.gz' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/conda/bin/NiChart_DLMUSE", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/main.py", line 250, in main run_pipeline( File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/dlmuse_pipeline.py", line 148, in run_pipeline apply_mask_img(df_img, in_dir, in_suff, mask_dir, mask_suff, out_dir, out_suff) File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/MaskImage.py", line 187, in apply_mask_img mask_img(in_img, in_mask, out_img) File "/opt/conda/lib/python3.10/site-packages/NiChart_DLMUSE/MaskImage.py", line 83, in mask_img nii_mask = nib.load(mask_img) File "/opt/conda/lib/python3.10/site-packages/nibabel/loadsave.py", line 103, in load raise FileNotFoundError(f"No such file or no access: '{filename}'") FileNotFoundError: No such file or no access: '/cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025/CapeTown/Protocols/DLMUSE/temp_working_dir/s2_dlicv/S4132_20110209_T1_DLICV.nii.gz Here is my code: singularity run --cleanenv -B /cbica/projects/PsychAnalysis/Pipelines/PHENOM_DLMUSE_2025:/pipelinedir/ nichart_dlmuse-1.0.10.sif -i ${in} -o ${out} -d cuda -c 1 Please let me know if you need more details. — Reply to this email directly, view it on GitHub<#82>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOHRPDRK2W75IOM4QYVZUIT3V3CVDAVCNFSM6AAAAACIHPWDIWVHI2DSMVQWIX3LMV43ASLTON2WKOZTGQ4DEMRUGY4DQMY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[AlexanderGetka-cbica]

Hi Dhivya,

I uploaded a new container version under the tag cbica/nichart_dlmuse:1.0.10-cuda11.8 .

Can you please try this one to see if it works directly? Please also make sure to run with --nv if using Singularity/Apptainer or --gpus all if running on Docker.

If that doesn't work we should make sure the modules are loaded as we expect. AFAIK, singularity can pick up the CUDA drivers from the node it's running on, so it might be as simple as putting "module load [cuda 12.1 module name]" in the SLURM script before running the original container, or similarly for cuda 11.8 with the alternative container.