🐛 Pipeline build freezes when 2 or more skullstripping options are enabled

[birajstha]

Describe the bug

This problem first occurred when I was testing this change.
After this change the ccs-options preconfig generates 3 different brain masks in the skull-stripping step with the following pre-config setting

anatomical_preproc:
  run: On

  brain_extraction:
    run: On

    # using: ['3dSkullStrip', 'BET', 'UNet', 'niworkflows-ants', 'FreeSurfer-ABCD', 'FreeSurfer-BET-Tight', 'FreeSurfer-BET-Loose', 'FreeSurfer-Brainmask']
    # this is a fork option
    using:
      - FreeSurfer-BET-Tight
      - FreeSurfer-BET-Loose
      - FreeSurfer-Brainmask

When trying to run this ccs-options pre-config, the pipeline build slowed during several stages like co-registration etc. and then freeze or killed eventually even just running test-config mode.
Later, I tested with a few different options for brain masks above. Each options of brain masking runs fine individually and hence it looks like a forking issue.

In general, when two or more options are enabled in the brain_extraction step during anatomical_preproc, then the pipeline building freezes, sometimes killed by SLURM out-of-memory handler.

I didn't test all preconfigs, but tested with 2 or more options in default and ccs-options.
Build did not freeze for anat-only preconfig and it ran fine without issues.

To reproduce

try running ccs-options preconfig after this change.

or,
Edit default preconfig and add 2 or more skull-stripping options, and try to run CPAC.

Preconfig

• default
• abcd-options
• anat-only
• blank
• ccs-options
• fmriprep-options
• fx-options
• monkey
• monkey-ABCD
• ndmg
• nhp-macaque
• preproc
• rbc-options
• rodent

Custom pipeline configuration

No response

Run command

No response

Expected behavior

'ccs-options` pipeline completes without errors and pipeline should not freeze when using 2 or more skull-stripping options.

Acceptance criteria

A full pipeline including both T1w and bold processing

• ccs-options preconfig completes without errors.
• build completes when 2 or more skull-stripping options are enabled.
• execution completes when 2 or more skull-stripping options are enabled.

Screenshots

C-PAC version

No response

Container platform

No response

Docker and/or Singularity version(s)

No response

Additional context

No response

[shnizzedy]

At a glance,

C-PAC/CPAC/anat_preproc/anat_preproc.py

Lines 1981 to 1996 in 266f7a4

	@nodeblock(
	name="brain_mask_acpc_freesurfer",
	switch=[
	["anatomical_preproc", "brain_extraction", "run"],
	["anatomical_preproc", "run"],
	],
	option_key=["anatomical_preproc", "brain_extraction", "using"],
	option_val="FreeSurfer-Brainmask",
	inputs=[
	"space-T1w_desc-brain_mask",
	"pipeline-fs_raw-average",
	"freesurfer-subject-dir",
	],
	outputs=["space-T1w_desc-acpcbrain_mask"],
	)
	def brain_mask_acpc_freesurfer(wf, cfg, strat_pool, pipe_num, opt=None):

this NodeBlock is most suspicious to me since it has the same config option as

C-PAC/CPAC/anat_preproc/anat_preproc.py

Lines 1960 to 1975 in 266f7a4

	@nodeblock(
	name="brain_mask_freesurfer",
	switch=[
	["anatomical_preproc", "brain_extraction", "run"],
	["anatomical_preproc", "run"],
	],
	option_key=["anatomical_preproc", "brain_extraction", "using"],
	option_val="FreeSurfer-Brainmask",
	inputs=[
	"pipeline-fs_raw-average",
	"pipeline-fs_brainmask",
	"freesurfer-subject-dir",
	],
	outputs=["space-T1w_desc-brain_mask"],
	)
	def brain_mask_freesurfer(wf, cfg, strat_pool, pipe_num, opt=None):

and has space-T1w_desc-brain_mask as an input