Junction filter fails on memory limit but does not try to resubmit with increased memory

[stanleyrc]

Description of the bug

Junction filtering for tumor only with hg38 fails due to memory using the default 72GB. The bug is that nextflow does not resubmit. After trying to resume multi times, the same job with 72GB was submitted and failed. Checking the job id with sacct shows that job ran out of memory.

[stanleyrc]

This was resolved by changing the initial memory parameters for JUNCTION_FILTER but this still seems like a bug that it was not resubmitted with increased memory. It seems like the R OOM error is not being passed to nextflow as OOM.

Temporary Fix:

Changed this in conf/base.config:

withName: 'JUNCTION_FILTER' {
    cpus            = { check_max( 2 * task.attempt, 'cpus' ) }
    memory          = { check_max( 72.GB * task.attempt, 'memory' ) }
    time            = { check_max( 8.h  * task.attempt, 'time'    ) }
}

to this:

withName: 'JUNCTION_FILTER' {
    cpus            = { check_max( 2 * task.attempt, 'cpus' ) }
    memory          = { check_max( 120.GB * task.attempt, 'memory' ) }
    time            = { check_max( 8.h  * task.attempt, 'time'    ) }
}

[shihabdider]

This is most likely because it's trying to request 72*2 GB of memory, which might get the request auto-rejected. The memory blowup itself is coming from the ra.overlaps method, which does a O(n^2) search. I'll work on trying to optimize this.