Skip to content

Resource allocations for several tools are excessive #310

@spvensko

Description

@spvensko

Description of the bug

Disclaimer: I'm a long time Nextflow user, but new nf-core user, so maybe I'm missing something obvious.

I recently ran the cutandrun workflow for someone in my lab. I eventually got outputs, but also got a sternly worded e-mail from the sysadmins regarding my memory usage. Apparently several of the tools in the workflow are defined as process_high when they do not need to be.

For example, TRIMGALORE is requesting 72 gigabytes of memory, but rarely uses more than 4 gigabytes of memory for my samples.

TRIMGALORE gets the process_high label here: https://github.com/nf-core/cutandrun/blob/master/modules/local/for_patch/trimgalore/main.nf#L3.
The process_high label is defined here: https://github.com/nf-core/cutandrun/blob/master/conf/base.config#L41-L45.

The other offenders seem to be BOWTIE2_SPIKEIN_ALIGN, BOWTIE2_TARGET_ALIGN, BOWTIE2_BUILD_SPIKEIN, and BOWTIE2_BUILD_TARGET.

I assume many users are not running into this due to the default profiles (e.g. -profile singularity) setting global CPU and memory caps. I, instead, used my personal cluster-specific profile which does not set any sort of global CPU or memory caps.

Command used and terminal output

Relevant files

spvensko_per_job_underused.tsv

System information

Nextflow version: 24.04.3.5916
Hardware: HPC
Executor: SLURM
Container engine: Singularity
OS: Linux
Version of nf-core/cutandrun: 3.2.2

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions