Comparing results across runs for routine genomic surveillance

[Leobpfbac]

Hi,
I would like to use MAGMA for genomic surveillance. After running the full pipeline for several batches, is there a way to run only the transmission-related functions for new batches without reprocessing everything from the beginning and still compare them to the previous runs?

[abhi18av]

Hi @Leobpfbac ,

Thanks for this question!

is there a way to run only the transmission-related functions for new batches without reprocessing everything from the beginning and still compare them to the previous runs?

This is an interesting question - could you please elaborate more on the use-case? I'm trying to understand why the typical resume functionality wouldn't suffice for this work.

NOTE: As this is a question, I've moved this to the Discussion board rather than the issues. If any code-related changes pop-up, I'll create the relevant issues and link them here for tracking :)

[Leobpfbac]

Hi @abhi18av! Thank you for getting back to me so quickly.
I’m considering a scenario where I run MAGMA every week, and then once a month I want to combine all the previous analyses to compare SNP distances and phylogeny across all samples processed so far.
How can I merge results from multiple MAGMA runs to compare samples across batches, for example, SNP distance, phylogeny and resistance?
Should I prepare a new samplesheet that includes all sample information?
What should I do with the work files generated for each batch to retain those results for future analyses?
I understand that the resume will complete or rerun features missing from the previous run, but I’m not sure how to merge several batches so that I can generate a combined snp_dist matrix and a single phylogeny including all samples.
What is the recommended procedure to accomplish this without needing to rerun the entire pipeline from scratch each time?

Sorry if this is a very basic question, but I really haven’t been able to figure out a practical way to do this.
Thank you very much for your time.

[abhi18av]

Hmm, interesting use-case @Leobpfbac - thanks for the outline.

Pinging Davi, as he might have a more up to date idea of the setup at INI - @Mxrcon , do you have any thoughts regarding this?

I think it should be possible to just reuse the generated GVCF file and then use that in the next run (with newer FASTQs), instead of passing the FASTQ files for the previous batches.

[Leobpfbac]

I was thinking about using the gVCF files from the previous analyses, but how can I prepare the samplesheet using these files, since the pipeline expects to start with FASTQ.gz files? Thanks @abhi18av and @Mxrcon.

[Mxrcon]

Hi all, jumping in since we have a very similar use case at Instituto Evandro Chagas.

In our routine setup, we process sequencing batches and later combine them for broader analyses (SNP distance, phylogeny, resistance, etc.). What has worked well for us is using the -resume option together with an ever-growing samplesheet. We simply append new samples to the same samplesheet (keeping all parameters and the same work/ directory).
With this approach, MAGMA reuses cached results from previous batches and processes only the new FASTQs, allowing us to maintain a unified dataset without re-running everything from scratch. So I strongly agree with @abhi18av on leveraging the existing caching mechanism.

That said, I also think Abhinav’s suggestion about working directly with gVCFs becomes especially valuable when the dataset grows larger (e.g., >100 samples). Reusing existing gVCFs could drastically reduce runtime and storage, and it aligns with the idea of treating them as a “variant database.”
This is something that would worth to explore for projects like GEMIBRA and REVIGET, where gVCFs can be stored centrally and imported as needed without triggering new variant calling.

A possible improvement to the pipeline could be:

• an option to export the gVCFs produced by MAGMA in a clean, organized structure;

• and a complementary option to import custom gVCFs (together with lineage-specific reference gVCFs) directly into a new run.

This would enable a hybrid mode: regular FASTQ processing for new samples, plus gVCF ingestion for previous batches or external datasets.

Happy to help test or draft ideas if this becomes a feature request.

[Leobpfbac]

OK, @abhi18av and @Mxrcon. I hope you’re able to implement gVCF support. I think it would make things easier, faster to run, and would require less storage space. I’ll proceed as you suggested. Thanks for the guidance.