Issues running NiBetaSeries with Singularity

[Tsjitsjikow]

Hello,

Hopefully I am posting my issues/questions in the right place - since they are very specific to NiBetaSeries it seems appropriate to post them here. Alternatively, I can re-post at e.g. Neurostars.

Unfortunately I am stuck, trying to run NiBetaSeries using Singularity (on HPCs). In our HPC environment we have personal and a shared, more powerful (main) VM. Up until now we have used singularity to run fmriprep (successfully) on both VMs.

After pulling NiBetaSeries image on my personal VM, I receive the following error when attempting to run:

Traceback (most recent call last):
File "/opt/miniconda-latest/bin/nibs", line 8, in
sys.exit(main())
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/cli/run.py", line 163, in main
from ..workflows.base import init_nibetaseries_participant_wf
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/base.py", line 26, in
from .model import init_betaseries_wf
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/model.py", line 16, in
from nistats import version as nistats_ver
File "/home/leovb/.local/lib/python3.7/site-packages/nistats/init.py", line 32, in
import nilearn
File "/home/leovb/.local/lib/python3.7/site-packages/nilearn/init.py", line 67, in
_check_module_dependencies()
File "/home/leovb/.local/lib/python3.7/site-packages/nilearn/version.py", line 118, in _check_module_dependencies
install_info=module_metadata.get('install_info'))
File "/home/leovb/.local/lib/python3.7/site-packages/nilearn/version.py", line 64, in _import_module_with_version_check
module = import(module_name)
ModuleNotFoundError: No module named 'joblib'. Module "joblib" could not be found. See http://nilearn.github.io/introduction.html#installation for installation information.

This seems strange to me, because the singularity image should contain all that is required?

When trying to run the same image on the main VM, I get stuck at the following:

/opt/miniconda-latest/lib/python3.7/site-packages/bids/layout/models.py:157: UserWarning: Accessing entities as attributes is deprecated as of 0.7. Please use the .entities dictionary instead (i.e., .entities['task'] instead of .task.
% (attr, attr))
/opt/miniconda-latest/lib/python3.7/site-packages/bids/layout/models.py:157: UserWarning: Accessing entities as attributes is deprecated as of 0.7. Please use the .entities dictionary instead (i.e., .entities['space'] instead of .space.
% (attr, attr))
Traceback (most recent call last):
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/utils.py", line 300, in _exec_query
res = layout.get(**query)[0]
IndexError: list index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/miniconda-latest/bin/nibs", line 8, in
sys.exit(main())
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/cli/run.py", line 282, in main
work_dir=work_dir,
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/base.py", line 160, in init_nibetaseries_participant_wf
description=description_label)
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/utils.py", line 276, in collect_data
for modality, query in file_queries.items()}
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/utils.py", line 276, in
for modality, query in file_queries.items()}
File "/opt/miniconda-latest/lib/python3.7/site-packages/nibetaseries/workflows/utils.py", line 303, in _exec_query
raise Exception(msg) from e
Exception: could not find file matching these criteria: {'datatype': 'func', 'subject': '02', 'task': 'nnip', 'run': None, 'session': None, 'suffix': 'regressors', 'desc': 'confounds', 'extension': '.tsv'}

In both cases, I attempt to run NiBetaSeries using the following shell script:

#!/bin/bash
LC_ALL=C

INDIR="/data/leo/NNIP/NNIP_Bids/"
OUTDIR="/data/leo/NNIP/NNIP_Bids/derivatives"
WORKDIR="/data/leo/working"
IMAGE="/data/leo/nibetaseries/nibetaseries-v0.6.0.simg" 

singularity run --cleanenv -B /data -B /run/shm:/run/shm $IMAGE \
                            nibs -c csf \
                                  --participant-label 02 \
                                  -w $WORKDIR \
                                  $INDIR \
                                  fmriprep \
                                  $OUTDIR \
                                  participant`

Which is a modification from the script we use for fmriprep.

As far as I know, the data is structured correctly. fmriprep preprocessing has been done with version 20.2.0. All data consists of a single run/session and single task.

Ideally I would be able to run NiBetaSeries on my personal VM (thus solving the joblib issue), but I imagine I would eventually run into the the same error as on the main VM here, so that may be a better one to tackle first?

Any help or thoughts are very much appreciated.

Leo

[Tsjitsjikow]

Hello,

Providing an update, since I made some progress.

RE: the issue on VM1, with singularity and joblib: I installed the nibetaseries Python module, and ran into the same issue as described for VM2:

Exception: could not find file matching these criteria: {'datatype': 'func', 'subject': '02', 'task': 'nnip', 'run': None, 'session': None, 'suffix': 'regressors', 'desc': 'confounds', 'extension': '.tsv'}

The following issue was very helpful: #303. This made me re-check the folder/file structure of my data.

fMRIprep 20.2.0 outputs 'sub-05_task-nnip_desc-confounds_timeseries.tsv' ; copying this file and changing the name of the copy to 'sub-05_task-nnip_desc-confounds_regressors.tsv' got me past this error.

Currently the workflow starts - but I run into memory issues - will briefly update if I can run fully successfully.

Apologies if this issue is now a conglomerate of issues - I can edit/adjust if desired, kindly let me know.

Leo

[Tsjitsjikow]

Switching from LSS to LSA solved the memory issue, and I can successfully obtain the beta series.

The memory issue that occurs with LSS as estimator is most likely due to the functional scans being in a single run/session, and relatively long (45minutes), and (only) 20gb available.

Please let me know if I should close this issue - I am also happy to troubleshoot the other issues or attempt to help otherwise.

Best wishes,

Leo

[Comeani]

Hi Leo, I am running into this same issue you had with the confounds .tsv file name while attempting to run nibetaseries in a singularity container on my university HPC. The inputs I'm using are from fMRIPrep 20.2.5.

It looks like a newer version of the BIDS specification prompted the change from confounds_regressors to confounds_timeseries.
https://neurostars.org/t/naming-change-confounds-regressors-to-confounds-timeseries/17637

Not sure how many others are trying to use this tool, but perhaps the change was missed?

[Tsjitsjikow]

Hi Comeani, exactly as you say. Hopefully everyone using NiBetaSeries and running into the same issue, will at least find this post and be able to proceed :-)

[Comeani]

Instead of going your route and changing the file name, I forked the repo and manually edited these code sections to resolve the issue (at least with my data as input):

NiBetaSeries/src/nibetaseries/conftest.py

Lines 75 to 83 in aa7d2ea

	deriv_regressor_entities = {
	**subject_entities,
	"datatype": "func",
	"task": "waffles",
	"run": 1,
	"description": "confounds",
	"suffix": "regressors",
	"extension": "tsv",
	}

NiBetaSeries/src/nibetaseries/workflows/utils.py

Lines 64 to 66 in aa7d2ea

	'confounds': _combine_dict(preproc_dict_ns, {'suffix': 'regressors',
	'desc': 'confounds',
	'extension': '.tsv'}),

I changed the suffix from 'regressors' to 'timeseries' on a new branch, and now running my nibs command from the terminal is successful. I have not yet re-run the tox tests specified in the Making Contributions tutorial, perhaps someone better acquainted with the code (@jdkent maybe?) can inform us if the fix is more complicated than this. I wanted to detail things here more before springing a pull request on folks.

I'm still learning much about Singularity containers, but I'm guessing seeing this change in the image is more involved than merging changes with the source here...