Skip to content

Fix multiple workflow in subworkflow #3486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 18 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 20 additions & 15 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- Add nf-core template version badges to README ([#3396](https://github.com/nf-core/tools/pull/3396))
- Add Bluesky badge to readme ([#3475](https://github.com/nf-core/tools/pull/3475))
- Run awsfulltest after release, and with dev revision on PRs to master ([#3485](https://github.com/nf-core/tools/pull/3485))
- Separate `PIPELINE_COMPLETION` in its own file ([#3486](https://github.com/nf-core/tools/pull/3486))

### Linting

Expand All @@ -16,33 +17,37 @@
- Fix: linting with comments after the input directive ([#3458](https://github.com/nf-core/tools/pull/3458))
- EDAM ontology fixes ([#3460](https://github.com/nf-core/tools/pull/3460))
- Fix default linting of nf-core components when `nf-core pipelines lint` is ran ([#3480](https://github.com/nf-core/tools/pull/3480))
- Fix components linting in `nf-core subworkflows lint` with multiple workflows in a `main.nf` ([#3486](https://github.com/nf-core/tools/pull/3486))

### Modules

- increase meta index for multiple input channels ([#3463](https://github.com/nf-core/tools/pull/3463))
- Increase meta index for multiple input channels ([#3463](https://github.com/nf-core/tools/pull/3463))
- Configure the default module repository, branch, and path from environment variables. ([#3481](https://github.com/nf-core/tools/pull/3481))

### Subworkflows

- Forbid more than one `workflow` block by `main.nf` ([#3486](https://github.com/nf-core/tools/pull/3486))

### General

- output passed to write_params_file as Path object ([#3435](https://github.com/nf-core/tools/pull/3435))
- chore(deps): update python:3.12-slim docker digest to 69ce3ae ([#3433](https://github.com/nf-core/tools/pull/3433))
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.4 ([#3438](https://github.com/nf-core/tools/pull/3438))
- format name/value with YAML syntax ([#3442](https://github.com/nf-core/tools/pull/3442))
- chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker.python to v3.2.0 ([#3446](https://github.com/nf-core/tools/pull/3446))
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.5 ([#3445](https://github.com/nf-core/tools/pull/3445))
- chore(deps): update pre-commit hook pre-commit/mirrors-mypy to v1.15.0 ([#3447](https://github.com/nf-core/tools/pull/3447))
- Output passed to write_params_file as Path object ([#3435](https://github.com/nf-core/tools/pull/3435))
- Chore(deps): update python:3.12-slim docker digest to 69ce3ae ([#3433](https://github.com/nf-core/tools/pull/3433))
- Chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.4 ([#3438](https://github.com/nf-core/tools/pull/3438))
- Format name/value with YAML syntax ([#3442](https://github.com/nf-core/tools/pull/3442))
- Chore(deps): update pre-commit hook editorconfig-checker/editorconfig-checker.python to v3.2.0 ([#3446](https://github.com/nf-core/tools/pull/3446))
- Chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.5 ([#3445](https://github.com/nf-core/tools/pull/3445))
- Chore(deps): update pre-commit hook pre-commit/mirrors-mypy to v1.15.0 ([#3447](https://github.com/nf-core/tools/pull/3447))
- Update prettier to 3.5.0 ([#3448](https://github.com/nf-core/tools/pull/3448))
- chore(deps): update python:3.12-slim docker digest to 34656cd ([#3450](https://github.com/nf-core/tools/pull/3450))
- Chore(deps): update python:3.12-slim docker digest to 34656cd ([#3450](https://github.com/nf-core/tools/pull/3450))
- Remove Twitter from README ([#3454](https://github.com/nf-core/tools/pull/3454))
- docs: fix contributing link in the main README ([#3459](https://github.com/nf-core/tools/pull/3459))
- Docs: fix contributing link in the main README ([#3459](https://github.com/nf-core/tools/pull/3459))
- Cleanup: Removed Redundant if Condition ([#3468](https://github.com/nf-core/tools/pull/3468))
- chore(deps): update gitpod/workspace-base docker digest to 7f35e40 ([#3473](https://github.com/nf-core/tools/pull/3473))
- chore(deps): update python:3.12-slim docker digest to aaa3f8c ([#3474](https://github.com/nf-core/tools/pull/3474))
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.9 ([#3470](https://github.com/nf-core/tools/pull/3470))
- chore(deps): update github actions ([#3488](https://github.com/nf-core/tools/pull/3488))
- chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.11.0 ([#3492](https://github.com/nf-core/tools/pull/3492))
- Chore(deps): update gitpod/workspace-base docker digest to 7f35e40 ([#3473](https://github.com/nf-core/tools/pull/3473))
- Chore(deps): update python:3.12-slim docker digest to aaa3f8c ([#3474](https://github.com/nf-core/tools/pull/3474))
- Chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.9.9 ([#3470](https://github.com/nf-core/tools/pull/3470))
- Chore(deps): update github actions ([#3488](https://github.com/nf-core/tools/pull/3488))
- Chore(deps): update pre-commit hook astral-sh/ruff-pre-commit to v0.11.0 ([#3492](https://github.com/nf-core/tools/pull/3492))
- Remove duplicated `pdiff` in `requirements.txt` ([#3486](https://github.com/nf-core/tools/pull/3486))

## [v3.2.0 - Pewter Pangolin](https://github.com/nf-core/tools/releases/tag/3.2.0) - [2025-01-27]

Expand Down
2 changes: 1 addition & 1 deletion nf_core/pipeline-template/main.nf
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
include { {{ short_name|upper }} } from './workflows/{{ short_name }}'
{%- if modules %}
include { PIPELINE_INITIALISATION } from './subworkflows/local/utils_nfcore_{{ short_name }}_pipeline'
include { PIPELINE_COMPLETION } from './subworkflows/local/utils_nfcore_{{ short_name }}_pipeline'
include { PIPELINE_COMPLETION } from './subworkflows/local/utils_nfcore_pipeline_completion'
{%- if igenomes %}
include { getGenomeAttribute } from './subworkflows/local/utils_nfcore_{{ short_name }}_pipeline'

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
//
// Subworkflow with for pipeline completion
//

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IMPORT FUNCTIONS / MODULES / SUBWORKFLOWS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/

{% if nf_schema %}
include { paramsSummaryMap } from 'plugin/nf-schema'
{% endif %}
{%- if email %}
include { completionEmail } from '../../nf-core/utils_nfcore_pipeline'
{%- endif %}
include { completionSummary } from '../../nf-core/utils_nfcore_pipeline'
{%- if adaptivecard or slackreport %}
include { imNotification } from '../../nf-core/utils_nfcore_pipeline'
{%- endif %}

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SUBWORKFLOW FOR PIPELINE COMPLETION
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/

workflow PIPELINE_COMPLETION {

take:
{%- if email %}
email // string: email address
email_on_fail // string: email address sent on pipeline failure
plaintext_email // boolean: Send plain-text email instead of HTML
{%- endif %}
outdir // path: Path to output directory where results will be published
monochrome_logs // boolean: Disable ANSI colour codes in log output
{%- if adaptivecard or slackreport %}
hook_url // string: hook URL for notifications
{% endif %}
{%- if multiqc %}
multiqc_report // string: Path to MultiQC report
{% endif %}

main:
{%- if nf_schema %}
summary_params = paramsSummaryMap(workflow, parameters_schema: "nextflow_schema.json")
{%- else %}
summary_params = [:]
{%- endif %}

{%- if multiqc %}
def multiqc_reports = multiqc_report.toList()
{%- endif %}

//
// Completion email and summary
//
workflow.onComplete {
{%- if email %}
if (email || email_on_fail) {
completionEmail(
summary_params,
email,
email_on_fail,
plaintext_email,
outdir,
monochrome_logs,
{% if multiqc %}multiqc_reports.getVal(),{% else %}[]{% endif %}
)
}
{%- endif %}

completionSummary(monochrome_logs)

{%- if adaptivecard or slackreport %}
if (hook_url) {
imNotification(summary_params, hook_url)
}
{%- endif %}
}

workflow.onError {
log.error "Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/subworkflows/yaml-schema.json
name: "pipeline_completion"
description: Notify user of pipeline completion
keywords:
- notification
- email
- completion
components: []
input:
- email:
type: string
description: Email address
- email_on_fail:
type: string
description: Email address sent on pipeline failure
- plaintext_email:
type: boolean
description: Send plain-text email instead of HTML
- outdir:
type: string
description: Path to output directory where results will be published
- monochrome_logs:
type: boolean
description: Disable ANSI colour codes in log output
- hook_url:
type: string
description: Hook URL for notifications
- multiqc_report:
type: string
description: Path to MultiQC report
output:
- dummy_emit:
type: boolean
description: |
Dummy emit to make nf-core subworkflows lint happy
authors:
- "@mirpedrol"
maintainers:
- "@mirpedrol"
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,7 @@
*/

{% if nf_schema %}include { UTILS_NFSCHEMA_PLUGIN } from '../../nf-core/utils_nfschema_plugin'
include { paramsSummaryMap } from 'plugin/nf-schema'
include { samplesheetToList } from 'plugin/nf-schema'{% endif %}
{%- if email %}
include { completionEmail } from '../../nf-core/utils_nfcore_pipeline'
{%- endif %}
include { completionSummary } from '../../nf-core/utils_nfcore_pipeline'
{%- if adaptivecard or slackreport %}
include { imNotification } from '../../nf-core/utils_nfcore_pipeline'
{%- endif %}
include { UTILS_NFCORE_PIPELINE } from '../../nf-core/utils_nfcore_pipeline'
include { UTILS_NEXTFLOW_PIPELINE } from '../../nf-core/utils_nextflow_pipeline'

Expand Down Expand Up @@ -112,70 +104,6 @@ workflow PIPELINE_INITIALISATION {
versions = ch_versions
}

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SUBWORKFLOW FOR PIPELINE COMPLETION
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*/

workflow PIPELINE_COMPLETION {

take:
{%- if email %}
email // string: email address
email_on_fail // string: email address sent on pipeline failure
plaintext_email // boolean: Send plain-text email instead of HTML
{%- endif %}
outdir // path: Path to output directory where results will be published
monochrome_logs // boolean: Disable ANSI colour codes in log output
{%- if adaptivecard or slackreport %}
hook_url // string: hook URL for notifications{% endif %}
{%- if multiqc %}
multiqc_report // string: Path to MultiQC report{% endif %}

main:
{%- if nf_schema %}
summary_params = paramsSummaryMap(workflow, parameters_schema: "nextflow_schema.json")
{%- else %}
summary_params = [:]
{%- endif %}

{%- if multiqc %}
def multiqc_reports = multiqc_report.toList()
{%- endif %}

//
// Completion email and summary
//
workflow.onComplete {
{%- if email %}
if (email || email_on_fail) {
completionEmail(
summary_params,
email,
email_on_fail,
plaintext_email,
outdir,
monochrome_logs,
{% if multiqc %}multiqc_reports.getVal(),{% else %}[]{% endif %}
)
}
{%- endif %}

completionSummary(monochrome_logs)

{%- if adaptivecard or slackreport %}
if (hook_url) {
imNotification(summary_params, hook_url)
}
{%- endif %}
}

workflow.onError {
log.error "Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting"
}
}

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FUNCTIONS
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# yaml-language-server: $schema=https://raw.githubusercontent.com/nf-core/modules/master/subworkflows/yaml-schema.json
name: "pipeline_initialisation"
description: Initialise the pipeline parameters, validate them and prepare the samplesheet
keywords:
- initialisation
- validation
- parameters
components: []
input:
- version:
type: boolean
description: Display version and exit
- validate_params:
type: boolean
description: Boolean whether to validate parameters against the schema at runtime
- monochrome_logs:
type: boolean
description: Do not use coloured log outputs
- nextflow_cli_args:
type: array
description: List of positional nextflow CLI args
- outdir:
type: string
description: The output directory where the results will be saved
- input:
type: string
description: Path to input samplesheet
output:
- samplesheet:
description: Channel with the samplesheet
structure:
- meta:
type: map
description: Metadata map
- fastq:
type: File
description: Fastq file
pattern: "*.fastq"
Comment on lines +37 to +38
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the samplesheet can be csv, tsv, yamls or json

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but this is the output of the read samplesheet.
In the template it only contains the metamap and fastq files, no ?

- versions:
description: Channel containing software versions file
structure:
- versions.yml:
type: file
description: File containing versions of the software used
authors:
- "@mirpedrol"
maintainers:
- "@mirpedrol"
1 change: 1 addition & 0 deletions nf_core/pipelines/create/create.py
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,7 @@ def render_template(self) -> None:
rename_files: Dict[str, str] = {
"workflows/pipeline.nf": f"workflows/{short_name}.nf",
"subworkflows/local/utils_nfcore_pipeline_pipeline/main.nf": f"subworkflows/local/utils_nfcore_{short_name}_pipeline/main.nf",
"subworkflows/local/utils_nfcore_pipeline_pipeline/meta.yml": f"subworkflows/local/utils_nfcore_{short_name}_pipeline/meta.yml",
}

# Set the paths to skip according to customization
Expand Down
Loading
Loading