Heavy Memory Usage?

[fhopp]

Hi there,
I just found this package and was wondering about average memory usage?
I am trying to denoise a single subject and 32 GB RAM seem not enough, i.e., python throws me a MemoryError.
I just wanted to know if memory usage is simply very high or whehter there might be a leakage?
Thanks,
Freddy

[arielletambini]

Hi Freddy,
I haven't had trouble with excessive memory usage, but I imagine that this may occur if the data that you are using is relatively large, e.g. high resolution or multiband? I (and others) have used it with relatively standard EPI scans (with file sizes around a few hundred MBs), e.g. TR = 2 seconds, duration of ~10 mins, 3x3x3mm voxels. What is your data like?
Arielle

[fhopp]

Hi Arielle,

yeah the data is indeed multiband with a 0.76 TR and high res voxels. However, I was able to run it successfully on one of our larger machines (126GB RAM) and mem usage was around 40-50GB per subject. It executed smoothly, see command below. As you can see, I am generating log files that store the output of each subject, perhaps something we can integrate into the base code? I was also thinking about multiprocessing, but given the large nature of my data, this might only be helpful for smaller use cases.

I forked a copy and made a small edit to allow for a bit more intuitive parsing of the cols arguments. Perhaps (very likely :) ) I was just too stupid to figure out how to submit the arguments, but now you can just submit a comma separated list.

Anyways, THANK YOU for starting this. Works like a charm so far (except I am finding that not all motion params are completely regressed out, i.e., the DVARS after de-noising maintain (especially larger) artifacts. There is also: https://github.com/HBClab/NuisanceRegression which I havent tried yet as it does not include high pass filter.

Freddy

`%%bash

subs=(sub-02 sub-03 sub-04 sub-05 sub-06 sub-07 sub-08 sub-09 sub-10 sub-11 sub-12 sub-13 sub-14 sub-15 sub-16 sub-17 sub-18 sub-19 sub-20 sub-21 sub-22 sub-23 sub-24 sub-25 sub-26 sub-27 sub-28 sub-29 sub-30 sub-31 sub-32 sub-33 sub-34 sub-36 sub-37 sub-38 sub-39 sub-40 sub-41 sub-42 sub-43 sub-44 sub-45 sub-46 sub-47 sub-48 sub-49 sub-51 sub-52 sub-53 sub-54 sub-55 sub-56 sub-57 sub-59 sub-60 sub-61 sub-62)

for element in "${subs[@]}"; do
log="$element"_log.txt
python /usr/local/anaconda3/lib/python3.6/site-packages/denoiser/run_denoise.py
/srv/lab/fmri/mft/dicom4bids/bids_nii/derivatives/fmriprep/"$element"/func/"$element"_task-stories_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz
/srv/lab/fmri/mft/dicom4bids/bids_nii/derivatives/fmriprep/"$element"/func/"$element"_task-stories_desc-confounds_regressors.tsv
/srv/lab/fmri/mft/dicom4bids/bids_nii/derivatives/denoised/"$element"
--col_names=csf,white_matter,framewise_displacement,trans_x,trans_y,trans_z,rot_x,rot_y,rot_z --hp_filter=0.01
&>> "$element"_log.txt 2>&1; done `