synthseg crash
Summary
Single subject qsiprep runs smoothly up until mri_synthseg, and then the job gets killed.
Additional details
- QSIPrep version: 0.18.1
- Singularity version: apptainer version 1.3.0-1.el8
What were you trying to do?
I was trying to preprocess a single subject dataset with qsiprep.
What did you expect to happen?
I expected the command I gave to run smoothly because it has worked for other subjects.
What actually happened?
The dwi preprocessing gets done up but the synthseg part crashes.
Reproducing the bug
Here is the command:
apptainer run qsiprep-0.18.1.sif 03_data/ 03_data/derivatives participant --participant-label 43 --output_resolution 1.2 --fs-license-file license.txt --work-dir work --nthreads 4 --omp-nthreads 3 -v -v
Here is the crash report:
Node: qsiprep_wf.single_subject_43_wf.anat_preproc_wf.synthseg_anat_wf.synthseg
Working directory: ~/work/qsiprep_wf/single_subject_43_wf/anat_preproc_wf/synthseg_anat_wf/synthseg
Node inputs:
args = <undefined>
environ = {'OMP_NUM_THREADS': '1'}
fast = False
input_image = ~/work/qsiprep_wf/single_subject_43_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-43_T1w_lps_trans.nii.gz
num_threads = 1
out_post = <undefined>
out_qc = <undefined>
out_seg = <undefined>
robust = <undefined>
subjects_dir = <undefined>
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.
Cmdline:
mri_synthseg --i ~/work/qsiprep_wf/single_subject_43_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-43_T1w_lps_trans.nii.gz --threads 1 --post sub-43_T1w_lps_trans_post.nii.gz --qc sub-43_T1w_lps_trans_qc.csv --o sub-43_T1w_lps_trans_aseg.nii.gz
Stdout:
SynthSeg 2.0
using 1 thread
Stderr:
DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
Traceback (most recent call last):
File "/opt/freesurfer/bin/mri_synthseg", line 2581, in <module>
main()
File "/opt/freesurfer/bin/mri_synthseg", line 123, in main
predict(
File "/opt/freesurfer/bin/mri_synthseg", line 236, in predict
net = build_model(model_file_segmentation=path_model_segmentation,
File "/opt/freesurfer/bin/mri_synthseg", line 632, in build_model
net.load_weights(model_file_segmentation, by_name=True)
File "/usr/local/miniconda/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
File "/usr/local/miniconda/lib/python3.8/site-packages/tensorflow/python/keras/engine/network.py", line 1263, in load_weights
hdf5_format.load_weights_from_hdf5_group_by_name(
File "/usr/local/miniconda/lib/python3.8/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 730, in load_weights_from_hdf5_group_by_name
original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'
Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
setattr(outputs, key, val)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'out_seg' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '~/work/qsiprep_wf/single_subject_43_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-43_T1w_lps_trans_aseg.nii.gz' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 400, in run
outputs = self.aggregate_outputs(runtime)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '~/work/qsiprep_wf/single_subject_43_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-43_T1w_lps_trans_aseg.nii.gz' for output 'out_seg' of a SynthSeg interface
-->
I am getting the same error with qsiprep 1.0.1
This is my log:
250922-09:49:48,372 nipype.workflow IMPORTANT:
Running QSIPrep version 1.0.2.dev0+gfc89945.d20250405
License NOTICE ##################################################
QSIPrep 1.0.2.dev0+gfc89945.d20250405
Copyright The PennLINC Developers.
This product includes software developed by
the NiPreps Community (https://nipreps.org/).
Portions of this software were developed at the Department of
Psychology at Stanford University, Stanford, CA, US.
This software is also distributed as a Docker container image.
The bootstrapping file for the image ("Dockerfile") is licensed
under the MIT License.
This software may be distributed through an add-on package called
"Docker Wrapper" that is under the BSD 3-clause License.
#################################################################
250922-09:49:48,592 nipype.workflow IMPORTANT:
Building QSIPrep's workflow:
* BIDS dataset path: /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata.
* Participant list: ['SOSE8L5YX'].
* Run identifier: 20250922-094851_20eb278c-3e49-4e20-8eaf-6a810cef7c25.
250922-09:49:50,835 nipype.workflow IMPORTANT:
Collected data:
dwi:
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/dwi/sub-SOSE8L5YX_dwi.nii.gz
flair: []
fmap:
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/fmap/sub-SOSE8L5YX_acq-dwi_dir-AP_epi.nii.gz
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/fmap/sub-SOSE8L5YX_acq-dwi_dir-PA_epi.nii.gz
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/fmap/sub-SOSE8L5YX_dir-AP_epi.nii.gz
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/fmap/sub-SOSE8L5YX_dir-PA_epi.nii.gz
roi: []
sbref: []
t1w:
- /zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/anat/sub-SOSE8L5YX_run-1_T1w.nii.gz
t2w: []
250922-09:49:50,870 nipype.workflow INFO:
Running nonlinear normalization to template
250922-09:49:50,906 nipype.workflow INFO:
Grouping DWI scans
250922-09:49:50,940 nipype.workflow INFO:
Combining all DWI files within each available session
250922-09:49:50,954 nipype.workflow INFO:
- 1 scans in session None
250922-09:49:51,35 nipype.workflow INFO:
Found 1 groups of DWI series based on their warp spaces:
[ { 'concatenated_bids_name': 'sub-SOSE8L5YX',
'dwi_series': [ '/zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/dwi/sub-SOSE8L5YX_dwi.nii.gz'],
'dwi_series_pedir': 'j',
'fieldmap_info': {'suffix': None}}]
250922-09:49:51,36 nipype.workflow INFO:
Found 1 groups of DWI series that can be corrected by eddy:
[ { 'concatenated_bids_name': 'sub-SOSE8L5YX',
'dwi_series': [ '/zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/dwi/sub-SOSE8L5YX_dwi.nii.gz'],
'dwi_series_pedir': 'j',
'fieldmap_info': {'suffix': None}}]
250922-09:49:51,36 nipype.workflow INFO:
Finished grouping DWI scans
250922-09:49:51,36 nipype.workflow INFO:
[{'dwi_series': ['/zi/flstorage/group_csp/in_house_datasets/sosense/mri/rawdata/sub-SOSE8L5YX/dwi/sub-SOSE8L5YX_dwi.nii.gz'], 'fieldmap_info': {'suffix': None}, 'dwi_series_pedir': 'j', 'concatenated_bids_name': 'sub-SOSE8L5YX'}]
250922-09:49:51,371 nipype.workflow INFO:
Automatically using 5, 5, 5 window for dwidenoise
250922-09:49:51,390 nipype.workflow INFO:
Using 8 threads in eddy
250922-09:50:09,359 nipype.workflow INFO:
QSIPrep workflow graph with 244 nodes built successfully.
250922-09:51:02,525 nipype.workflow IMPORTANT:
QSIPrep started!
250922-09:51:52,109 nipype.workflow INFO:
[Node] Setting-up "qsiprep_1_0_wf.sub_SOSE8L5YX_wf.anat_preproc_wf.synthseg_wf.synthseg" in "/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg".
250922-09:51:52,172 nipype.workflow INFO:
[Node] Executing "synthseg" <qsiprep.interfaces.freesurfer.SynthSeg>
250922-09:52:05,80 nipype.interface WARNING:
Changing /zi/home/johannes.wiesner/work/slurm/project_sosense/tmp/633451/sub-SOSE8L5YX/anat/sub-SOSE8L5YX_space-ACPC_desc-brain_mask.nii.gz dtype from float64 to uint8
250922-09:52:06,62 nipype.workflow INFO:
[Node] Finished "synthseg", elapsed time 13.870578s.
250922-09:52:06,63 nipype.workflow WARNING:
Storing result file without outputs
250922-09:52:06,72 nipype.workflow WARNING:
[Node] Error on "qsiprep_1_0_wf.sub_SOSE8L5YX_wf.anat_preproc_wf.synthseg_wf.synthseg" (/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg)
250922-09:52:17,910 nipype.workflow ERROR:
Node synthseg failed to run on host zislhcn0111.zi.local.
250922-09:52:17,925 nipype.workflow ERROR:
Saving crash info to /zi/home/johannes.wiesner/work/slurm/project_sosense/tmp/633451/sub-SOSE8L5YX/log/20250922-094851_20eb278c-3e49-4e20-8eaf-6a810cef7c25/crash-20250922-095217-johannes.wiesner-synthseg-31940c34-e104-4a8e-95e7-eb1aefd20d54.txt
Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.
Cmdline:
mri_synthseg --cpu --i /zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/dl_prep_wf/resample_skulled_to_reference/sub-SOSE8L5YX_run-1_T1w_lps_trans.nii.gz --threads 1 --post sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz --qc sub-SOSE8L5YX_run-1_T1w_lps_trans_qc.csv --o sub-SOSE8L5YX_run-1_T1w_lps_trans_aseg.nii.gz
Stdout:
SynthSeg 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
Stderr:
Traceback (most recent call last):
File "/opt/freesurfer/bin/mri_synthseg", line 2581, in <module>
main()
File "/opt/freesurfer/bin/mri_synthseg", line 123, in main
predict(
File "/opt/freesurfer/bin/mri_synthseg", line 195, in predict
labels_segmentation, _ = get_list_labels(label_list=labels_segmentation)
File "/opt/freesurfer/bin/mri_synthseg", line 1446, in get_list_labels
label_list = np.array(reformat_to_list(label_list, load_as_numpy=True, dtype='int'))
File "/opt/freesurfer/bin/mri_synthseg", line 1534, in reformat_to_list
if isinstance(var, (int, float, np.int, np.int32, np.int64, np.float, np.float32, np.float64)):
File "/zi/home/johannes.wiesner/.local/lib/python3.10/site-packages/numpy/__init__.py", line 324, in __getattr__
raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 457, in aggregate_outputs
setattr(outputs, key, val)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 325, in validate
value = super().validate(objekt, name, value, return_pathlike=True)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'out_post' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg/sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 404, in run
outputs = self.aggregate_outputs(runtime)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 464, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg/sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz' for output 'out_post' of a SynthSeg interface
250922-09:55:17,548 nipype.interface WARNING:
Changing /zi/home/johannes.wiesner/work/slurm/project_sosense/tmp/633451/sub-SOSE8L5YX/dwi/sub-SOSE8L5YX_space-ACPC_desc-brain_mask.nii.gz dtype from float64 to uint8
250922-09:55:26,176 nipype.workflow ERROR:
could not run node: qsiprep_1_0_wf.sub_SOSE8L5YX_wf.anat_preproc_wf.synthseg_wf.synthseg
250922-09:55:26,448 nipype.workflow CRITICAL:
QSIPrep failed: Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.
Cmdline:
mri_synthseg --cpu --i /zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/dl_prep_wf/resample_skulled_to_reference/sub-SOSE8L5YX_run-1_T1w_lps_trans.nii.gz --threads 1 --post sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz --qc sub-SOSE8L5YX_run-1_T1w_lps_trans_qc.csv --o sub-SOSE8L5YX_run-1_T1w_lps_trans_aseg.nii.gz
Stdout:
SynthSeg 2.0
using CPU, hiding all CUDA_VISIBLE_DEVICES
using 1 thread
Stderr:
Traceback (most recent call last):
File "/opt/freesurfer/bin/mri_synthseg", line 2581, in <module>
main()
File "/opt/freesurfer/bin/mri_synthseg", line 123, in main
predict(
File "/opt/freesurfer/bin/mri_synthseg", line 195, in predict
labels_segmentation, _ = get_list_labels(label_list=labels_segmentation)
File "/opt/freesurfer/bin/mri_synthseg", line 1446, in get_list_labels
label_list = np.array(reformat_to_list(label_list, load_as_numpy=True, dtype='int'))
File "/opt/freesurfer/bin/mri_synthseg", line 1534, in reformat_to_list
if isinstance(var, (int, float, np.int, np.int32, np.int64, np.float, np.float32, np.float64)):
File "/zi/home/johannes.wiesner/.local/lib/python3.10/site-packages/numpy/__init__.py", line 324, in __getattr__
raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 457, in aggregate_outputs
setattr(outputs, key, val)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 325, in validate
value = super().validate(objekt, name, value, return_pathlike=True)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'out_post' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg/sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 404, in run
outputs = self.aggregate_outputs(runtime)
File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 464, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/zi/home/johannes.wiesner/work/slurm/project_sosense/work_dirs/qsiprep/qsiprep_1_0_wf/sub_SOSE8L5YX_wf/anat_preproc_wf/synthseg_wf/synthseg/sub-SOSE8L5YX_run-1_T1w_lps_trans_post.nii.gz' for output 'out_post' of a SynthSeg interface
This can be fixed by updating FreeSurfer in https://github.com/PennLINC/qsiprep_build/blob/main/Dockerfile_FreeSurfer The newer version fixes the numpy deprecation: https://github.com/freesurfer/freesurfer/commit/583a2390e59b240c9bbd9950e9f2d41428edc1bc
You can mount your own FreeSurfer (version 7.4.1) folder when running QSIprep in docker to fix this temporarily, like:
docker run --rm -u 1005:1005 -v ${data_dir}:/data \ -v ${output_dir}:/out \ -v ${work_dir}/qsiprep_temp_${subj}:/work \ -v /home/user/opt/freesurfer:/opt/freesurfer:ro \ -e FREESURFER_HOME=/opt/freesurfer \ pennlinc/qsiprep:1.0.2 /data/BIDS ...