Support for HPCs that do no allow --mem

[bigbigdumdum]

Hello!

I am using nfcore/rnaseq on an HPC which do not allow memory specification (SLURM --mem flag), and instead uses cpu counts to allocate memory to a job. I found workaround for all other cases via a additions to the config file as well as a custom submission script.

That is except in nf-core/rnaseq/modules/nf-core/qualimap/rnaseq/main.nf in line

def memory = (task.memory.mega*0.8).intValue() + 'M'

Here the process infers task.memory which is set to null in my config files to prevent nextflow from adding --mem flag in the SLURM script.

This breaks as expected with

-[nf-core/rnaseq] Pipeline completed with errors- ERROR ~ Error executing process > 'NFCORE_RNASEQ:RNASEQ:BAM_QC_RNASEQ:QUALIMAP_RNASEQ (<sample_id>)' Caused by: Cannot get property 'mega' on null object -- Check script '<user>/nextflow_cache/assets/nf-core/rnaseq/modules/nf-core/qualimap/rnaseq/main.nf' at line: 25 Source block: def args = task.ext.args ?: '' prefix = task.ext.prefix ?: "${meta.id}" def paired_end = meta.single_end ? '' : '-pe' def memory = (task.memory.mega*0.8).intValue() + 'M' def strandedness = 'non-strand-specific' if (meta.strandedness == 'forward') { strandedness = 'strand-specific-forward' } else if (meta.strandedness == 'reverse') { strandedness = 'strand-specific-reverse' } """ unset DISPLAY mkdir -p tmp export _JAVA_OPTIONS=-Djava.io.tmpdir=./tmp qualimap \\ --java-mem-size=$memory \\ rnaseq \\ $args \\ -bam $bam \\ -gtf $gtf \\ -p $strandedness \\ $paired_end \\ -outdir $prefix """

The the only fix that worked for me is to edit this line to

def memory = task.memory ? (task.memory.mega*0.8).intValue() + 'M' : (task.cpus * 4000) + 'M'

where it check where tast.memory is set, if so proceeds as usual, else uses a different option.
This however this is specific to my setup as my allocated memory is tied to cpus.
A generic solution would be.

def memory = task.memory ? (task.memory.mega*0.8).intValue() + 'M' : '4000M'

I know this is likely a rare case. But the solution does not change anything in the current workflow, while accommodating such edge cases.

Please make this small change. My config file with identifiable information removed is pasted in the end.

Thank you!

params.slurm_account = '<my account>'

workDir = '<workdir>'

env {
    TMPDIR = '/tmp/'
    TMP = '/tmp/'
    TEMP = '/tmp/'
}

process {
    executor       = 'slurm'
    queue          = 'std'
    beforeScript   = 'source /sw/batch/init.sh'
    memory         = {null}
    clusterOptions = "--nodes=1 --ntasks=1 --account=${params.slurm_account}"
    withName: '.*' {
        memory = { null }
    }
    // Forcefully overrides nf-core's internal ceiling calculations
    resourceLimits = [
        memory: null
    ]

    
    withName: 'NFCORE_RNASEQ:RNASEQ:.*:PICARD_MARKDUPLICATES' {
        ext.args = { "-Xmx${(task.cpus * 4000 * 0.8).intValue()}m" }
    }

    // errorStrategy = 'ignore'
    // maxRetries    = 3

    withLabel: process_single {
        cpus     = 8
        time     = 4.h
        maxForks = 21
        memory   = {null}
    }
    withLabel: process_low {
        cpus     = 8
        time     = 8.h
        maxForks = 21
        memory   = {null}
    }
    withLabel: process_medium {
        cpus     = 24
        time     = 12.h
        maxForks = 22
        memory   = {null}
    }
    withLabel: process_high {
        cpus     = 48
        time     = 12.h
        maxForks = 16
        memory   = {null}
    }
    withLabel: process_long {
        cpus     = 48
        time     = 12.h
        maxForks = 22
        memory   = {null}
    }
    withLabel: process_high_memory {
        cpus     = 48
        time     = 12.h
        maxForks = 12
        memory   = {null}
    }
}

executor {
    name            = 'slurm'
    queueSize       = 500
    submitRateLimit = '60/1min'
    submitTemplate  = "<run dir>/templates/autosubmit_template.sh"

}

apptainer {
    enabled    = true
    autoMounts = true
    cacheDir   ="<storage parition>/apptainer_cache'
    runOptions = "\
    --bind $launchDir:$launchDir \
    --bind <partitions>:<partitions> \
    --bind /tmp/:/tmp/"
}

report   { enabled = true; overwrite = true; file = "${params.outdir}/nextflow_report.html" }
timeline { enabled = true; overwrite = true; file = "${params.outdir}/nextflow_timeline.html" }
trace    { enabled = true; overwrite = true; file = "${params.outdir}/nextflow_trace.tsv" }

[pinin4fjords]

Thanks for the detailed report and repro config!

This isn't really a Nextflow or nf-core/rnaseq issue, it's a bug in the qualimap/rnaseq module itself (shared code pulled in from nf-core/modules via nf-core modules install). The module assumes task.memory is always set, but that's not guaranteed, your config is a legitimate way to suppress --mem on HPCs that allocate memory by CPU count instead.

nf-core/modules already has an established pattern for this exact situation, used in ~126 modules (e.g. picard/markduplicates):

def avail_mem = 3072
if (!task.memory) {
    log.info('[Qualimap RNAseq] Available memory not known - defaulting to 3GB. Specify process memory requirements to change this.')
} else {
    avail_mem = (task.memory.mega * 0.8).intValue()
}

Would you be up for opening a PR against nf-core/modules with this fix? Worth noting qualimap/bamqc and qualimap/bamqccram have the same unguarded line, so it'd be good to patch all three while you're in there. Once merged, we can pull the update into rnaseq via nf-core modules update.

It'd also help to flag the PR for review in the #request-review channel on the nf-core Slack, that's where module PRs tend to get picked up faster.

[bigbigdumdum]

Thank you! I will open a PR. However, I am new to nf-core. Is there some standard test that is to be done before this. If so can you point to some resources for the same.

edit: i have submitted a pull request adressing the same. nf-core/modules#12441 . So I am closing this.