Skip to content

Enhance documentation on version emission in Nextflow #3707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nvnieuwk wants to merge 3 commits into
base: docs-v2
Choose a base branch
from
+99 −52
Open

Enhance documentation on version emission in Nextflow #3707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
93334eb
Enhance documentation on version emission in Nextflow
nvnieuwk Jan 12, 2026
f946886
Merge branch 'docs-v2' into versions-topics-update-docs-v2
nvnieuwk Jan 22, 2026
064f355
[automated] Fix code linting
nf-core-bot Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 68 additions & 24 deletions sites/docs/src/content/docs/specifications/components/modules/general.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,16 +173,11 @@ Only if a tool reads the input multiple times, uncompress the file before runnin

## Emission of versions

The topic output qualifier in Nextflow collects outputs from multiple processes across a pipeline.
Use this feature in nf-core modules to collect version information from all tools without complex channel mixing logic.
See the [fastqc module](https://github.com/nf-core/modules/blob/0c47e4193ddde2c5edbc206b5420cbcbee5c9797/modules/nf-core/fastqc/main.nf#L16) as an example.
The topic output qualifier is a new feature in Nextflow that provides a streamlined approach to collecting outputs from multiple processes across a pipeline.
This feature is particularly useful for nf-core modules to collect version information from all tools used in a pipeline without the complex channel mixing logic that was previously required.
See the [fastqc module](https://github.com/nf-core/modules/blob/d5416e7fb4d202b26a8ec9ebd3d2756907a16ec9/modules/nf-core/fastqc/main.nf#L16) as an example.

:::warning
For modules that use the template process directive, they will currently continue to depend on the old approach with `versions.yml`.
The only difference is that they should also use the topic output qualifier to send the `versions.yml` file to the versions topic.
:::

:::tip{title="Tips for extracting the version string" collapse}
:::note{title="Tips for extracting the version string" collapse}

`sed{:bash}` is a powerful stream editor that can be used to manipulate the input text into the desired output.
Start by piping the output of the version command to `sed{:bash}` and try to select the line with the version number:
Expand All @@ -192,21 +187,69 @@ tool --version | sed '1!d'
```

- `sed '1!d'{:bash}` Extracts only line 1 of the output printed by `tools --version{:bash}`.
- The line to process can also be selected using a pattern instead of a number: `sed '/pattern/!d'{:bash}`. For example, `sed '/version:/!d'{:bash}`.
- The line to process can also be selected using a pattern instead of a number: `sed '/pattern/!d'{:bash}`, e.g. `sed '/version:/!d'{:bash}`.
- If the line extraction hasn't worked, then it's likely the version information is written to stderr, rather than stdout.
In this case capture stderr using `|&{:bash}` which is shorthand for `2>&1 |{:bash}`.
- `sed 's/pattern/replacement/'{:bash}` can be used to remove parts of a string. `.` matches any character, `+` matches 1 or more times.
- You can separate `sed{:bash}` commands using `;`. Often the pattern: `sed 'filter line ; replace string'{:bash}` is enough to get the version number.
- You can separate `sed{:bash}` commands using `;`. Often the pattern : `sed 'filter line ; replace string'{:bash}` is enough to get the version number.
- It is not necessary to use `echo`, `head`, `tail`, or `grep`.
- Use `|| true` for tools that exit with a non-zero error code: `command --version || true{:bash}` or `command --version | sed ... || true{:bash}`.

:::

Where applicable, each module command MUST emit one output per tool containing the process name, the tool name and the tool version, e.g.

```bash
output:
<other outputs>
tuple val("${task.process}"), val("fastqc"), eval("fastqc --version | sed -e 's/FastQC v//g'"), emit: versions_fastqc, topic: versions
tuple val("${task.process}"), val("samtools"), eval("samtools --version |& sed '1!d ; s/samtools //'"), emit: versions_samtools, topic: versions
```

resulting in, for instance,

```bash
["FASTQC", "fastqc", "0.11.9"] # FASTQC.out.versions_fastqc
["SAMTOOLS_VIEW", "samtools", "1.12"] # FASTQC.out.versions_samtools
```

All reported versions MUST be without a leading `v` or similar (i.e. must start with a numeric character), or for unversioned software, a Git SHA commit id (40 character hexadecimal string).

If the software is unable to output a version number on the command-line then the version can be directly specified using `val()` instead of `eval()`

Please include the accompanying comments above the software packing directives and beside the version output.

```groovy {4,14}
process TOOL {

...
// WARN: Version information not provided by tool on CLI. Please update version string below when bumping container versions.
conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://depot.galaxyproject.org/singularity/tool:0.9.1--pl526hc9558a2_3' :
'biocontainers/tool:0.9.1--pl526hc9558a2_3' }"

...

output:
tuple val("${task.process}"), val("tool"), val("0.9.1"), emit: versions_tool, topic: versions
// WARN: Version information not provided by tool on CLI. Please update this string when bumping container versions.

...
}
```

:::warning
For modules that use the template process directive, for now they will continue to depend on the old approach with versions.yml (see below).
The only difference is that they should also use the topic output qualifier to send the versions.yml file to the versions topic.
:::

:::note
For not yet converted modules, you will see a different approach for collecting versions. Even though the approach is deprecated, it is shown below for reference.
For not yet converted modules, you will see a different approach for collecting versions. Even though the approach is deprecated, we kept it below for reference.
:::

Where applicable, each module command MUST emit a file `versions.yml` containing the version number for each tool executed by the module, for example:
:::tip{title="Deprecated approach for emitting versions" collapse}
Where applicable, each module command MUST emit a file `versions.yml` containing the version number for each tool executed by the module, e.g.

```bash
cat <<-END_VERSIONS > versions.yml
Expand All @@ -224,20 +267,20 @@ resulting in, for instance,
samtools: 1.12
```

All reported versions MUST be without a leading `v` or similar (that is, must start with a numeric character), or for unversioned software, a Git SHA commit id (40 character hexadecimal string).
All reported versions MUST be without a leading `v` or similar (i.e. must start with a numeric character), or for unversioned software, a Git SHA commit id (40 character hexadecimal string).

A [HEREDOC](https://tldp.org/LDP/abs/html/here-docs.html) is used over piping into the versions file line-by-line to avoid accidentally overwriting the file.
The exit status of sub-shells evaluated within the HEREDOC is ignored, ensuring that a tool's version command does not erroneously terminate the module.
We chose a [HEREDOC](https://tldp.org/LDP/abs/html/here-docs.html) over piping into the versions file line-by-line as we believe the latter makes it easy to accidentally overwrite the file.
Moreover, the exit status of the sub-shells evaluated in within the HEREDOC is ignored, ensuring that a tool's version command does no erroneously terminate the module.

If the software is unable to output a version number on the command-line, manually specify a variable called `VERSION` to provide this information. For example, [homer/annotatepeaks module](https://github.com/nf-core/modules/blob/master/modules/nf-core/homer/annotatepeaks/main.nf).
If the software is unable to output a version number on the command-line then a variable called `VERSION` can be manually specified to provide this information e.g. [homer/annotatepeaks module](https://github.com/nf-core/modules/blob/master/modules/nf-core/homer/annotatepeaks/main.nf).

Include the accompanying comments above the software packing directives and beside the version string.
Please include the accompanying comments above the software packing directives and beside the version string.

```groovy {4,15,21}
process TOOL {

...
// WARN: Version information not provided by tool on CLI. Update version string below when bumping container versions.
// WARN: Version information not provided by tool on CLI. Please update version string below when bumping container versions.
conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'https://depot.galaxyproject.org/singularity/tool:0.9.1--pl526hc9558a2_3' :
Expand All @@ -248,7 +291,7 @@ conda "${moduleDir}/environment.yml"
script:
def args = task.ext.args ?: ''
def prefix = task.ext.prefix ?: "${meta.id}"
def VERSION = '0.9.1' // WARN: Version information not provided by tool on CLI. Update this string when bumping container versions.
def VERSION = '0.9.1' // WARN: Version information not provided by tool on CLI. Please update this string when bumping container versions.
"""
...

Expand All @@ -261,12 +304,13 @@ END_VERSIONS
}
```

If the HEREDOC cannot be used because the script is not bash, write the `versions.yml` directly. For example, [ascat module](https://github.com/nf-core/modules/blob/master/modules/nf-core/ascat/main.nf).
If the HEREDOC cannot be used because the script is not bash, the `versions.yml` MUST be written directly e.g. [ascat module](https://github.com/nf-core/modules/blob/master/modules/nf-core/ascat/main.nf).
:::

## Presence of when statement
### Presence of when statement

The `when` statement MUST NOT be changed in the process definition.
Supply `when` conditions using the `process.ext.when` directive in a configuration file instead.
The process definition MUST NOT change the `when` statement.
`when` conditions can instead be supplied using the `process.ext.when` directive in a configuration file.

```groovy
process {
Expand Down
12 changes: 7 additions & 5 deletions sites/docs/src/pages/docs/[subsection].astro
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,13 @@ import { sanitizeNfCoreLabels, addEntriesToSection } from "@utils/functions";
import SidebarSubNav from "@components/sidebar/SidebarSubNav.astro";

export async function getStaticPaths() {
return ["get-started", "users", "developers", "contributors", "specifications", "community", "nf-core-tools"].map((subsection) => ({
params: {
subsection: subsection,
},
}));
return ["get-started", "users", "developers", "contributors", "specifications", "community", "nf-core-tools"].map(
(subsection) => ({
params: {
subsection: subsection,
},
}),
);
}

const { subsection } = Astro.params;
Expand Down
47 changes: 24 additions & 23 deletions sites/docs/src/pages/docs/index.astro
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,19 +109,19 @@ const featuredPages = [
title: "What is nf-core?",
href: "/docs/get_started/nf-core",
icon: "octicon:play-16",
description: "Introduction to the nf-core project and community"
description: "Introduction to the nf-core project and community",
},
{
title: "Environment setup",
href: "/docs/get_started/environment_setup/overview",
icon: "octicon:gear-16",
description: "Set up your system to run nf-core pipelines"
description: "Set up your system to run nf-core pipelines",
},
{
title: "Run your first pipeline",
href: "/docs/get_started/run-your-first-pipeline",
icon: "octicon:terminal-16",
description: "Step-by-step guide to running an nf-core pipeline"
description: "Step-by-step guide to running an nf-core pipeline",
},
];

Expand Down Expand Up @@ -174,20 +174,19 @@ const mainSections = [
},
];


// Define training cards
const trainingCards = [
{
title: "Nextflow documentation",
href: "https://www.nextflow.io/docs/latest/index.html",
icon: "octicon:book-16",
description: "Official Nextflow language documentation and guides"
description: "Official Nextflow language documentation and guides",
},
{
title: "Nextflow and nf-core training",
href: "https://training.nextflow.io/latest/",
icon: "octicon:mortar-board-16",
description: "Hands-on training materials and workshops"
description: "Hands-on training materials and workshops",
},
];
---
Expand All @@ -209,15 +208,15 @@ const trainingCards = [
featuredEntries.map((page) => (
<a href={page.href} class="text-decoration-none g-col-12 g-col-md-6 g-col-lg-4">
<div class="card h-100 featured-card">
<div class="card-body">
<div class="featured-icon mb-2">
<Icon name={page.customIcon} />
</div>
<h3 class="h5 card-title mb-2">
{page.metadata?.data.title || page.label}
</h3>
<p class="card-description mb-0">{page.description}</p>
<div class="card-body">
<div class="featured-icon mb-2">
<Icon name={page.customIcon} />
</div>
<h3 class="h5 card-title mb-2">
{page.metadata?.data.title || page.label}
</h3>
<p class="card-description mb-0">{page.description}</p>
</div>
</div>
</a>
))
Expand All @@ -232,7 +231,10 @@ const trainingCards = [
<div class="grid gap-3">
{
mainSections.map((section) => (
<a href={section.sectionHref} class="text-decoration-none g-col-12 g-col-md-6 g-col-lg-4">
<a
href={section.sectionHref}
class="text-decoration-none g-col-12 g-col-md-6 g-col-lg-4"
>
<div class="card h-100 section-card">
<div class="card-body">
<div class="section-icon mb-2">
Expand All @@ -252,21 +254,20 @@ const trainingCards = [
<div class="mb-5">
<h2 class="section-heading mb-2">Additional resources</h2>
<p class="mb-4">
Learn Nextflow and nf-core through comprehensive documentation and hands-on training
materials.
Learn Nextflow and nf-core through comprehensive documentation and hands-on training materials.
</p>
<div class="grid gap-3">
{
trainingCards.map((card) => (
<a href={card.href} class="text-decoration-none g-col-12 g-col-md-6">
<div class="card h-100 featured-card">
<div class="card-body">
<div class="featured-icon mb-2">
<Icon name={card.icon} />
</div>
<h3 class="h5 card-title mb-2">{card.title}</h3>
<p class="card-description mb-0">{card.description}</p>
<div class="card-body">
<div class="featured-icon mb-2">
<Icon name={card.icon} />
</div>
<h3 class="h5 card-title mb-2">{card.title}</h3>
<p class="card-description mb-0">{card.description}</p>
</div>
</div>
</a>
))
Expand Down
Loading