High Failure Rate in CALL_WF:LOFREQ_CALL, Multiple Pipeline Processes Not Executing, Including STRUCTURAL_VARIANTS_ANALYSIS_WF (v2.1.0)

[MelakG13]

@abhi18av

@TimHHH

@LennertVerboven

I'm encountering multiple issues while running the MAGMA pipeline (v2.1.0) using Nextflow:

There is a high failure rate in the CALL_WF:LOFREQ_CALL process, with 1,020 failed tasks out of 2,724 succeeded tasks. The pipeline ignores these errors and completes, but this failure rate is concerning.
Several downstream processes in the pipeline are not executing at all (marked with [-] in the logs), such as CALL_WF:LOFREQ_FILTER, MINOR_VARIANTS_ANALYSIS_WF, and MERGE_WF:PREPARE_COHORT_VCF. This suggests the pipeline is stalling after the CALL_WF:LOFREQ_CALL failures, despite the errors being ignored.
Specifically, in the STRUCTURAL_VARIANTS_ANALYSIS_WF workflow, processes after BWA_MEM__DELLY (which completes successfully) are not executed at all, including SAMTOOLS_MERGE__DELLY, DELLY_CALL, and others. This indicates a potential dependency or configuration issue in this workflow.

Environment Setup

I installed all the necessary environments as specified in the readme:

*magma-env-1
*magma-env-2
*magma-ntmprofiler-env
*magma-tbprofiler-env

These environments are located at /home/melak_g/mambaforge/envs/, as specified in my parameters file.

I ran the pipeline with the following command:

nextflow run 'https://github.com/TORCH-Consortium/MAGMA' -profile conda_local,server -r v2.1.0 -params-file my_parameters.yml

Here’s the content of my my_parameters.yml file:

input_samplesheet: /mnt/DATADRIVE0/TB_PROJECT/BIAY/MAGMA/samplesheet.csv
only_validate_fastqs: false
outdir: /mnt/DATADRIVE0/TB_PROJECT/BIAY/MAGMA/results
conda_envs_location: /home/melak_g/mambaforge/envs/

I attached the screenshots. I am asking for help.

[abhi18av]

Hi @MelakG13 ,

It's quite tough to diagnose this without looking at the .nextflow.log file and the configuration but here's my initial hunch.

I think the problem might be with

1. The quality of the samples - maybe the joint cohort stats file can reveal something.

2. The setup of lofreq tool - we do recommend using a container system to try and minimize possible hardware issues.

Perhaps check with the test profile first to make sure the pipeline setup is fine and then, I suggest you share the .nextflow.log file for a deeper look.

nextflow run TORCH-Consortium/MAGMA -profile docker,laptop,test

[MelakG13]

Hi @abhi18av
Thank you for being so supportive.

As per your suggestion, I first ran the pipeline using the test profile (nextflow run TORCH-Consortium/MAGMA -profile docker,laptop,test) and it was successful without any issues.

However, when running the pipeline with my samples, I encountered errors that made the run unsuccessful.

Some additional details:

I used the original params.yml file provided in the GitHub repository without modifications.
I reduced the sample size and ran the following command:

nextflow run TORCH-Consortium/MAGMA -profile docker,laptop -params-file my_parameters.yml -r v2.1.0 -resume

(I used -resume because there was a power interruption during the process.)

Despite this, the joint.merged_cohort_stats file shows zero for all columns except for raw total sequence.

To assist with the troubleshooting:

I have attached the .nextflow.log file for your review.
I have also attached screenshots showing the issue with the joint.merged_cohort_stats file

I appreciate your support!

.nextflow.log

[abhi18av]

Hi @MelakG13 , sorry the late response here. We've had a few days off in SA :)

I'll take a look soon and circle back.

[abhi18av]

Despite this, the joint.merged_cohort_stats file shows zero for all columns except for raw total sequence.

Is there any chance that I can download these samples from the public internet and try to reproduce this on my side?

[MelakG13]

@abhi18av Thanks for your response. My samples are unpublished, so they’re not publicly available. I confirm joint.merged_cohort_stats shows zeros except raw total sequence, with BAMs showing no mapped reads (samtools stats: reads mapped: 0). MultiQC shows 73–74 bp reads, likely failing BWA_MEM -k 100. I attached the screenshot of it, if it helps. Can you confirm if -k 100 is too strict for short reads? Can you advise how to set the parameter file in this case?

As an example, I just attached the samtool stats reads of one of my samples.

[abhi18av]

Mmm, I think you're right. The issue is possibly related to the k-mer length

As part of #216, I added a bwa_k66 profile, which you can enable using the -profile parameter.

Ideally I'd suggest doing this experiment with fewer samples to see if they have a better performance now.

[MelakG13]

I tried running that pipeline, and it worked well overall. However, some processes were not fully executed, such as drug resistance profiling, lineage classification, clustering results, and phylogenetic tree construction.

I used the following command:
nextflow run https://github.com/TORCH-Consortium/MAGMA -r v2.1.0 -profile docker,server,bwa_k66 --input_fastqs /mnt/DATADRIVE0/TB_PROJECT/BIAY/Aynu/Magma/fastq

At the end of the run, it showed that 38 tasks were not executed despite the pipeline completing.

[TimHHH]

Hey @MelakG13

Good to see that adjusting the k parameter helped.

The fact that the processes that are applied to merged sample dataset (e.g. phylogeny, clustering, profiling) fail is an inidicator that that there is something up with the software setup now rather than data from specific samples. These processes also use more complex and diverse softwares, @abhi18av I am guessing this may be the environment?

[MelakG13]

@TimHHH @abhi18av Thanks for your help! I found magma-container-1:2.0.0 lacks tbprofiler, bcftools, fasttree for MAGMA v2.1.0 (tested via docker run --rm -it ghcr.io/torch-consortium/magma/magma-container-1:2.0.0 /bin/bash). python3 3.10.8, java 11.0.15 are present. Is there a newer image or a fix?

[abhi18av]

Hi @MelakG13 ,

Actually we have split up the containers recently since we wish to keep frequently updated tools (i.e. tbprofiler), easy to upgrade, by using separate containers.

Here's an example

MAGMA/conf/docker.config

Line 39 in ee4fece

'.*TBPROFILER.*' {

I'd be happy to connect for quick meeting and try to speed up the analysis - feel free to book a slot here