Hanging on `fsnative2t1w_xfm`

[joey-scanga]

Describe the bug

Please correct me if this isn't permitted, but I'm crossposting from nipreps/fmriprep#3562 since there hasn't been any responses to that issue yet.

I'm consistently experiencing hanging in fmriprep when running with the --fs-no-resume flag whenever it reaches smriprep's fsnative2t1w_xfm node.

Exact command line executed

docker run --rm -it -u 2582:10 \
    -v /usr/local/pkg/freesurfer7.3/.license:/opt/freesurfer/license.txt:ro \
    -v /data/sylvester/data1/datasets/spc_new/rawdata:/data:ro \
    -v /data/sylvester/data1/datasets/spc_new/derivatives/fmriprep:/out \
    -v /data/sylvester/scratch1/spc_new/fmriprep/sub-SPC026_ses-01:/work \
    -v /data/sylvester/data1/datasets/spc_new/derivatives/fmriprep/sourcedata/freesurfer:/deriv1 \
    nipreps/fmriprep:25.2.2 \
    /data /out participant \
    --participant-label SPC026 \
    -w /work \
    --verbose \
    --fs-subjects-dir /deriv1 \
    --fd-spike-threshold 0.9 \
    --fs-no-resume \
    --cifti-output 91k \
    --output-spaces MNI152NLin2009cAsym

Are you positive that the input dataset is BIDS-compliant?

• I have used the online BIDS-Validator
• I have run a local installation of the BIDS-Validator (please, note the version of the validator here).
  Whichever version used in fmriprep 23.2.2 (the Dockerfile implies this is 1.14.10)
• I let sMRIPrep check it for me (in other words, I didn't set the --skip-bids-validation argument).
• No, I haven't checked myself AND used the --skip-bids-validation argument.

sMRIPrep feedback information
Please attach the full log written to the standard output and the crashfile(s), if generated.

Visual reports of sMRIPrep
If applicable, add screenshots or attach the corresponding "reportlets" to help explain your problem.

Installation type (please complete the following information):

• "Bare-metal"
• Singularity
• Docker

Additional context
Add any other context about the problem here.

Here's a screenshot of where fmriprep hangs at.

[effigies]

When you look at top or similar, are you seeing a mri_robust_register process running? If not, there's probably an error higher up in the log that crashed out fsnative2t1w_xfm.

[joey-scanga]

Good question... I'll have to run another instance and see, our filesystem is down atm but I'll update as soon as I can. Thanks!!

[joey-scanga]

This is the error I notice:

251203-19:56:40,322 nipype.workflow WARNING:
         [Node] Error on "fmriprep_25_2_wf.sub_SPC026_ses_01_wf.anat_fit_wf.surface_recon_wf.fsnative2t1w_xfm" (/work/fmriprep_25_2_wf/sub_SPC026_ses_01_wf/anat_fit_wf/surface_recon_wf/fsnative2t1w_xfm)
251203-19:56:40,329 nipype.workflow INFO:
         [Node] Setting-up "fmriprep_25_2_wf.sub_SPC026_ses_01_wf.anat_fit_wf.brain_extraction_wf.norm" in "/work/fmriprep_25_2_wf/sub_SPC026_ses_01_wf/anat_fit_wf/brain_extraction_wf/norm".
251203-19:56:40,335 nipype.workflow INFO:
         [Node] Executing "norm" <niworkflows.interfaces.fixes.FixHeaderRegistration>
exception calling callback for <Future at 0x7b68f927c350 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/engine/nodes.py", line 720, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/interfaces/freesurfer/base.py", line 149, in run
    return super().run(**inputs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/interfaces/base/core.py", line 392, in run
    self._check_mandatory_inputs()
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/interfaces/base/core.py", line 277, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: PatchedRobustRegister requires a value for input 'source_file'. For a list of required inputs, see PatchedRobustRegister.help()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.pixi/envs/fmriprep/lib/python3.12/concurrent/futures/process.py", line 264, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
                       ^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/engine/nodes.py", line 221, in result
    return _load_resultfile(
           ^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/engine/utils.py", line 290, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /work/fmriprep_25_2_wf/sub_SPC026_ses_01_wf/anat_fit_wf/surface_recon_wf/fsnative2t1w_xfm/result_fsnative2t1w_xfm.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.pixi/envs/fmriprep/lib/python3.12/concurrent/futures/_base.py", line 340, in _invoke_callbacks
    callback(self)
  File "/app/.pixi/envs/fmriprep/lib/python3.12/site-packages/nipype/pipeline/plugins/multiproc.py", line 174, in _async_callback
    result = args.result()
             ^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/.pixi/envs/fmriprep/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
FileNotFoundError: /work/fmriprep_25_2_wf/sub_SPC026_ses_01_wf/anat_fit_wf/surface_recon_wf/fsnative2t1w_xfm/result_fsnative2t1w_xfm.pklz

[ksreeniv]

@joey-scanga Were you able to find a workaround or solution for this?

[effigies]

Sorry, lost track of this. It looks like the get_surfaces node could not find T1 images. What do you see in /work/fmriprep_25_2_wf/sub_SPC026_ses_01_wf/anat_fit_wf/surface_recon_wf/get_surfaces/_report/report.rst? And what's the contents of your FreeSurfer directory? If you're using datalad, are the data all present? Does running datalad unlock help?