Description of feature
I've been testing oncoanalyzer on some data that includes ~120X tumour genomes.
To align these, the default process_high cpus of 12 * task.attempt is far too little for bwa_mem, causing it to fail at the 16 hour mark, and delaying processing by about a day while it re-runs with 24 cpus.
But when I set the cpus for process_high to 36 * task.attempt, this gives 36 cpus to esvee, which seems to scale up its RAM usage with the number of cpus. It hit around 280GB of RAM used (on retry 2 with 216GB requested) before being killed by the scheduler. Even with 12 cpus it needed around 143GB of RAM and one retry.
This creates a less than ideal situation of duelling resource requirements between these two processes with quite different needs but the same resource label. Bwa-mem needs much more CPU, but could get by with even less RAM, while esvee needs much more RAM, but could get by with less CPU.
A good possible solution might be to change esvee to the process_medium label (default 6 cpus), possibly adding in the process_high_mem label to increase the RAM.
For now I'm having to fiddle around with manually setting the cpus/RAM for esvee by name, which is not entirely nf-core best practice.
Thanks!
Description of feature
I've been testing oncoanalyzer on some data that includes ~120X tumour genomes.
To align these, the default
process_highcpus of12 * task.attemptis far too little for bwa_mem, causing it to fail at the 16 hour mark, and delaying processing by about a day while it re-runs with 24 cpus.But when I set the cpus for
process_highto36 * task.attempt, this gives 36 cpus to esvee, which seems to scale up its RAM usage with the number of cpus. It hit around 280GB of RAM used (on retry 2 with 216GB requested) before being killed by the scheduler. Even with 12 cpus it needed around 143GB of RAM and one retry.This creates a less than ideal situation of duelling resource requirements between these two processes with quite different needs but the same resource label. Bwa-mem needs much more CPU, but could get by with even less RAM, while esvee needs much more RAM, but could get by with less CPU.
A good possible solution might be to change esvee to the
process_mediumlabel (default 6 cpus), possibly adding in theprocess_high_memlabel to increase the RAM.For now I'm having to fiddle around with manually setting the cpus/RAM for esvee by name, which is not entirely nf-core best practice.
Thanks!