New pipeline: nf-core/midas

[MartinaCardinali]

Pipeline title/name

microbiome_differential_abudance_analyses

Keywords

microbiome, differential abundance, consensus, simulation, benchmarking

What is it about?

A microbiome differential abundance (DA) pipeline suited for 16S/metagenomic phyloseq data. It combines results from multiple DA tools into a single consensus call by two modes: (Path A) a _k-_intersection consensus that reports taxa called differentially abundant by at least k out of n tools, and (Path B) a simulation-trained soft consensus that uses MIDASim to generate ground-truth datasets, scores each tool's performance and applies an optimally weighted threshold to the results from the real data.

Please provide a schematic diagram of the proposed pipeline

What would a minimal first release of this pipeline include?

The pipeline is built entirely in R. General-purpose libraries for data handling and visualization: phyloseq, dplyr, tidyr, tibble, readr, stringr, purrr, ggplot2, ggrepel, openxlsx, optparse, jsonlite, fs. Simulation: MIDASim. Differential abundance tools: ADAPT, corncob, LinDA, LOCOM, and MaAsLin2 (metagenomeSeq is also included but disabled by default). All dependencies are bundled in a versioned Docker image (rocker/r-ver:4.5.1 base), which is also usable via Singularity.

I confirm my proposed pipeline will follow nf-core guidelines. Most importantly, my pipeline will:

• be built with Nextflow.
• pass nf-core lint tests and use standardized parameters.
• be community-owned and developed within the nf-core organization.
• open source under the MIT license with proper credits and acknowledgments.
• have a descriptive, all lowercase, and without punctuation name.
• use the nf-core pipeline template and predominantly use official nf-core modules.
• focus on a specific data/analysis type with appropriate scope.
• have properly maintained documentation.
• be bundled using versioned Docker/Singularity containers.

Why do we need a new pipeline?

It is well established that individual DA tools produce substantially different results on the same dataset making tool choice a major source of variability in microbiome studies. This pipeline addresses this by running five complementary DA tools (ADAPT, corncob, LinDA, LOCOM, MaAsLin2) in parallel on the same phyloseq input and combining their outputs into a single consensus call. Its simulation-trained soft consensus mode — where MIDASim generates ground-truth datasets from the user's own control samples, each tool is scored against that truth, and an optimally weighted threshold is applied to the real data — is, to our knowledge, not available in any existing pipeline or R package within the nf-core ecosystem.

Who would be interested?

Microbiome researchers performing differential abundance analysis on 16S or metagenomic data. This includes:

Computational biologists and bioinformaticians working on case-control microbiome studies who need a reproducible, tool-agnostic consensus call rather than relying on a single DA method.
Wet-lab microbiome researchers who want a turnkey pipeline that handles the full DA workflow — from phyloseq input to a final list of consensus DA taxa — without requiring expertise in each individual statistical tool.
Benchmarking and methods developers interested in evaluating DA tool performance on simulated data derived from real microbial communities (via MIDASim), or in comparing consensus strategies (k-intersection vs. simulation-trained soft consensus).

What has been done so far

The pipeline is fully implemented and functional. Both execution paths have been validated end-to-end on public datasets from MicrobiomeBenchmarkData (bacterial vaginosis, sub- and supragingival plaque). The codebase includes: a main workflow (workflows/midas.nf), 6 DA tool modules (each with simulated and real variants), subworkflows for simulation, scoring, and consensus, 12 R scripts in bin/, a versioned Dockerfile, and Docker/Singularity profiles.

nf-core scaffolding is in place: manifest block, .nf-core.yml, nextflow_schema.json, MIT license, CITATIONS.md, README.md, usage and output documentation, and versions.yml emission in all 17 modules.

What remains after proposal acceptance: running nf-core create to adopt the full template skeleton, wiring up nf-validation, setting up nf-test with a minimal test profile and public test data, CI via GitHub Actions, and publishing the container to a public registry.

The URL to the existing work is not given since the repository is not public yet; it will be transferred to the nf-core GitHub organisation upon acceptance.

URL to existing work (if applicable)

No response

Are there any similar existing nf-core pipelines?

there is the differentialabundance pipeline which has been developed for transcriptomics data, so not relevant for microbiome data.

[nf-core-bot]

Approval status: 🕐 Pending

Required approvals: Either 2 core team members OR 1 core team member + 1 maintainer

Review Status	Team members
✅ Approved (Core)	@jfy133
🕐 Pending (Core)	@ewels, @maxulysse, @mribeirodantas, @mashehu, @JoseEspinosa, @mirpedrol, @matthdsm, @FriederikeHanssen, @kenibrewer, @atrigila, @edmundmiller, @awgymer, @SPPearce, @sateeshperi, @FranBonath, @LouisLeNezet, @christopher-hakkaart, @nvnieuwk
🕐 Pending (Maintainer)	@pontus, @maxulysse, @lpantano, @pinin4fjords, @mashehu, @JoseEspinosa, @mirpedrol, @GallVp, @FloWuenne, @matthdsm, @FriederikeHanssen, @adamrtalbot, @mahesh-panchal, @jfy133, @atrigila, @prototaxites, @ramprasadn, @edmundmiller, @luisas, @SPPearce, @camlloyd, @vagkaratzas, @sateeshperi, @jonasscheid, @famosab, @nictru, @itrujnara, @LouisLeNezet, @Joon-Klaps, @jasmezz, @Felix-Kummer, @sofstam, @nvnieuwk

[itrujnara]

We have discussed it before, and I think the idea is solid. We just need to work on the name, since "midas" is not descriptive enough, and anything including "differential abundance" could be too similar to, well, differentialabundance. Speaking of the latter, the purposes feel similar, but as it stands the two are entirely disjoint, so I'm not sure they can be integrated without creating too much chaos (although an opinion from Ionas would be valuable). Since I have participated in creating the proposal, I will abstain from voting.

[jfy133]

Have'nt had a chance to read the proposal yet (but I promise to do so, please ping me for thoughts if you don't hear from me within a week!), but note there are lots of metagenomics pipelines called MIDAS, famoulsy: https://github.com/snayfach/MIDAS

So yeah wouldn't be suitable regardless of descprtiveness

[itrujnara]

For the name, I would propose differentialmetagenome, should get the purpose across

[jfy133]

Hi @MartinaCardinali

You say:

there is the differentialabundance pipeline which has been developed for transcriptomics data, so not relevant for microbiome data.

But I think that is not 💯 correct in the sense that even though the examples are all transcriptomic and there a couple of RNA specific tools, all the inputs are actually pretty generic (just tabular).

e.g. you can run DeSeqR on microbiome/metagenomic hits just fine.

If we ignored the current focus on the transcriptomics in differentialabundance, would there be other blockers/differences that would not fit in that pipeline that warrants a seaprate pipeline?

@pinin4fjords your thoughts/feedback would also be helpful here too :)

[pinin4fjords]

@jfy133 Fair point - and propd (already in differentialabundance) is compositional too, so "DA is transcriptomics-only" doesn't quite hold. DA is already multi-method (DESeq2/limma/edgeR/propd off one input), so a k-of-n consensus over those isn't a crazy fit in principle.

But the novel bit here is the simulation-trained soft consensus (MIDASim ground truth, per-tool scoring, optimal weighting) - a whole second execution model with no analogue in DA, and microbiome-specific. With the phyloseq input and the compositional tool bundle DA doesn't carry, I think it warrants its own pipeline rather than a mode bolted onto DA. If the plain k-intersection consensus later proves useful generically, that's a candidate to lift into a shared module.

[MartinaCardinali]

Hi both,

Thanks for the thoughtful feedback — good points about differentialabundance being input-agnostic in principle.

Already @pinin4fjords addressed the novelty given by the path B of the pipeline.

A few things that justify a separate pipeline:

Path A was thought for users who do not want to perform simulations. The k-intersection consensus logic and the microbiome-specific tools are missing from the differentialabundance pipeline, and the fit feels forced for what is a different analytical question (agreement across methods, not just differential abundance from a single method).

The tool selection is deliberate: the statistical models are complementary, not redundant. The tools in the pipeline (ADAPT, corncob, LinDA, LOCOM, MAsLin2) were chosen because they rely on different underlying statistical frameworks: Tobit censored regression that treats zeros as left-censored observations (ADAPT), beta-binomial regression (corncob), linear models on log-ratio transforms (LinDA), logistic regression on compositional data with permutation-based inference (LOCOM), generalized linear and mixed models with multiple normalizations (MAsLin2). This complementarity is what makes the consensus meaningful: agreement across methods with different assumptions. This is a microbiome-specific design choice — the tool bundle in differentialabundance (DESeq2, limma, edgeR) was designed for RNA-seq count data and lacks microbiome-specific compositional corrections.

In summary, the two paths are complementary, not redundant. Path A gives users a quick, interpretable multi-tool consensus without any simulation overhead. Path B gives users a statistically grounded, data-driven consensus when they want higher confidence. Having both in one pipeline lets users choose the level of rigor appropriate for their study, while sharing the same tool infrastructure and input format.

[jfy133]

OK, thank you both for the feedback @pinin4fjords and the more detail @MartinaCardinali!

You have a

/approve

from me once we get a better name :)

Maybe something like differentialabundancemicrobiome even (long though).

I do have a few further comments though:

The pipeline is built entirely in R. General-purpose libraries for data handling and visualization: phyloseq, dplyr, tidyr, tibble, readr, stringr, purrr, ggplot2, ggrepel, openxlsx, optparse, jsonlite, fs. Simulation: MIDASim.

Does this mean a lot of custom R scripts? You will need to be careful to keep each one very simple where possible.

All dependencies are bundled in a versioned Docker image (rocker/r-ver:4.5.1 base), which is also usable via Singularity.

This will be a problem, we don't one one massive docker image but (ideally) one container process with just the dependencies needed for that particular step.

That said I think it would be quite easy to make Seqera containers given all CRAN packages are on conda-forge (e.g. the tidyverse ones) and all of the bioconductor packages are on bioconda and thus have biocontainers (e.g. phyloseq).

[MartinaCardinali]

Great to hear @jfy133!

Regarding the name, how about damicrobiome or microbiomeda to keep it shorter and with the acronym just for differential abundance? Or even difabmicrobiome.

On the R scripts: the DA tool scripts are thin wrappers around the corresponding R/Bioconductor packages. The scoring and simulation scripts are more substantial as they implement the core methodology, but they are self-contained and each handle a single task.

On the containers: we'll move to per-process containers. Some of our dependencies are already on bioconda (corncob, MaAsLin2) or conda-forge (MicrobiomeStat/LinDA), so Seqera Containers would work for those. However, not all CRAN/Bioconductor packages have conda recipes yet. LOCOM is GitHub-only, and MIDASim (CRAN) and ADAPT (Bioconductor) don't appear to have conda-forge/bioconda packages. We'll need to either contribute conda recipes or use custom Dockerfiles for those processes. We'll sort this out as part of the nf-core adaptation.