Description of the bug
Disclaimer: I'm a long time Nextflow user, but new nf-core user, so maybe I'm missing something obvious.
I recently ran the cutandrun workflow for someone in my lab. I eventually got outputs, but also got a sternly worded e-mail from the sysadmins regarding my memory usage. Apparently several of the tools in the workflow are defined as process_high when they do not need to be.
For example, TRIMGALORE is requesting 72 gigabytes of memory, but rarely uses more than 4 gigabytes of memory for my samples.
TRIMGALORE gets the process_high label here: https://github.com/nf-core/cutandrun/blob/master/modules/local/for_patch/trimgalore/main.nf#L3.
The process_high label is defined here: https://github.com/nf-core/cutandrun/blob/master/conf/base.config#L41-L45.
The other offenders seem to be BOWTIE2_SPIKEIN_ALIGN, BOWTIE2_TARGET_ALIGN, BOWTIE2_BUILD_SPIKEIN, and BOWTIE2_BUILD_TARGET.
I assume many users are not running into this due to the default profiles (e.g. -profile singularity) setting global CPU and memory caps. I, instead, used my personal cluster-specific profile which does not set any sort of global CPU or memory caps.
Command used and terminal output
Relevant files
spvensko_per_job_underused.tsv
System information
Nextflow version: 24.04.3.5916
Hardware: HPC
Executor: SLURM
Container engine: Singularity
OS: Linux
Version of nf-core/cutandrun: 3.2.2
Description of the bug
Disclaimer: I'm a long time Nextflow user, but new nf-core user, so maybe I'm missing something obvious.
I recently ran the cutandrun workflow for someone in my lab. I eventually got outputs, but also got a sternly worded e-mail from the sysadmins regarding my memory usage. Apparently several of the tools in the workflow are defined as
process_highwhen they do not need to be.For example,
TRIMGALOREis requesting 72 gigabytes of memory, but rarely uses more than 4 gigabytes of memory for my samples.TRIMGALOREgets theprocess_highlabel here: https://github.com/nf-core/cutandrun/blob/master/modules/local/for_patch/trimgalore/main.nf#L3.The
process_highlabel is defined here: https://github.com/nf-core/cutandrun/blob/master/conf/base.config#L41-L45.The other offenders seem to be
BOWTIE2_SPIKEIN_ALIGN,BOWTIE2_TARGET_ALIGN,BOWTIE2_BUILD_SPIKEIN, andBOWTIE2_BUILD_TARGET.I assume many users are not running into this due to the default profiles (e.g.
-profile singularity) setting global CPU and memory caps. I, instead, used my personal cluster-specific profile which does not set any sort of global CPU or memory caps.Command used and terminal output
Relevant files
spvensko_per_job_underused.tsv
System information
Nextflow version: 24.04.3.5916
Hardware: HPC
Executor: SLURM
Container engine: Singularity
OS: Linux
Version of nf-core/cutandrun: 3.2.2