Skip to content

Template upgrade#143

Merged
muffato merged 39 commits intofirst_precomp_buscofrom
template_upgrade
Mar 14, 2025
Merged

Template upgrade#143
muffato merged 39 commits intofirst_precomp_buscofrom
template_upgrade

Conversation

@muffato
Copy link
Copy Markdown
Member

@muffato muffato commented Mar 7, 2025

Closes #136 and #137
Supersedes #140 and #141
Affects #142

This is my work so far on upgrading the nf-core template of this pipeline. I've done up to 3.1. Still need to do 3.2 and then a round of polishing.

@DLBPointon . Through the template upgrades I felt compelled to fully adopt nf-validation/nf-schema, i.e. use it for samplesheet parsing as well (previously, nf-validation was only use to check the parameters).
I didn't do it in subworkflows/local/utils_nfcore_blobtoolkit_pipeline/main.nf as suggested though because the pipeline needs to support two flavours of samplesheets, and the fetchngs one needs an extra process, CAT_CAT, which I wasn't sure had its place in PIPELINE_INITIALISATION.
I also included your fix re. calling MINIMAP2

@tkchafin . Because the template suggests passing all input data through channels in the two workflows of main.nf and subworkflows/local/utils_nfcore_blobtoolkit_pipeline/main.nf and workflows/blobtoolkit.nf. I've changed the interface to pass all databases through 1 channel. In doing so, I also had to fix a few things like adding .first() etc like you did.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

muffato and others added 21 commits March 3, 2025 14:20
@muffato
Template update for nf-core/tools version 2.12.1
111b596
@muffato
Template upgrade (2.12)
439ff4f
@muffato
Template update for nf-core/tools version 2.13.1
b872503
@muffato
bugfix: the version is in word 4
9501c97
@muffato
Report the full paths rather than the staging names
8876890
@muffato
Include the scheme (https, etc)
10deaaf
@muffato
Not needed since the channels goes through the download/untar process
96538eb
@muffato
bugfix for the handling of busco=null
dd7c609
@muffato
For safety, let's make these value channels
d2326fb
@muffato
Template upgrade (2.13)
97978b8
@muffato
error is the recommended method
789ea60
@muffato
Make the subworkflow support queue channels
5f93113
@DLBPointon @muffato
Added missing parameters
2f1c02d
@muffato
Template update for nf-core/tools version 2.14.1
dc8c62b
@muffato
Merge branch 'TEMPLATE' into dev
987cf4b
@muffato
Template update for nf-core/tools version 3.0.2
210efdc
@muffato
Merge branch 'TEMPLATE' into dev
0dc14d7
@muffato
Template update for nf-core/tools version 3.1.2
7ab4dab
@muffato
Merge branch 'TEMPLATE' into dev
42ab54b
@muffato
Test data are not under assets/ any more
21b0254
@muffato
Updated the licence years
e94e07d
@muffato muffato requested review from DLBPointon and tkchafin March 7, 2025 09:19
@muffato muffato self-assigned this Mar 7, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 7, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit e7f8d57

+| ✅ 179 tests passed       |+
#| ❔  28 tests were ignored |#
!| ❗  25 tests had warnings |!
Details

❗ Test warnings:

  • files_exist - File not found: conf/igenomes_ignored.config
  • nextflow_config - Config variable not found: validation.help.beforeText
  • nextflow_config - Config variable not found: validation.help.afterText
  • nextflow_config - Config variable not found: validation.summary.beforeText
  • nextflow_config - Config variable not found: validation.summary.afterText
  • nextflow_config - Config manifest.version should end in dev: 0.7.0
  • pipeline_todos - TODO string in ro-crate-metadata.json: "description": "# sanger-tol/blobtoolkit\n\nGitHub Actions CI Status\nGitHub Actions Linting StatusCite with Zenodo\nnf-test\n\nNextflow\nrun with conda\nrun with docker\nrun with singularity\nLaunch on Seqera Platform\n\n## Introduction\n\nsanger-tol/blobtoolkit is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)\n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run sanger-tol/blobtoolkit \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\n## Credits\n\nsanger-tol/blobtoolkit was originally written by priyanka-surana.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use sanger-tol/blobtoolkit for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX \n\n Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nThis pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license.\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",
  • schema_description - Ungrouped param in schema: busco_output
  • local_component_structure - create_bed.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - nohit_list.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - restructurebuscodir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - generate_config.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - compressblobdir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - windowstats_input.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
  • local_component_structure - run_blastx.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - view.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - collate_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - run_blastn.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - minimap_alignment.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - prepare_genome.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - coverage_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - finalise_blobdir.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - input_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - blobtools.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
  • local_component_structure - busco_diamond_blastp.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

  • files_exist - File is ignored: CODE_OF_CONDUCT.md
  • files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: conf/igenomes.config
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
  • template_strings - template_strings
  • merge_markers - merge_markers

✅ Tests passed:

Run details

  • nf-core/tools version 3.2.0
  • Run at 2025-03-12 18:00:06

DLBPointon
DLBPointon previously approved these changes Mar 7, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@DLBPointon DLBPointon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me,

You have a couple of FIXME's in [workflows/blobtoolkit.nf](https://github.com/sanger-tol/blobtoolkit/pull/143/files#diff-7a28bedb4a4d0b2a2efbecbed161f99b0de22d4313b14418623b8b9a220d942b) are these remnants or still need work?

@DLBPointon DLBPointon self-requested a review March 7, 2025 10:14
@DLBPointon DLBPointon dismissed their stale review March 7, 2025 10:14

Tests failing

@DLBPointon
Copy link
Copy Markdown
Contributor

DLBPointon commented Mar 7, 2025 •
edited
Loading

The process.shell may have to be set inside of the module that needs it in a more explicit way, UNTAR and GUNZIP may need patching in that case.
- I'm just testing it now

Fixed by 3.2 template

@DLBPointon DLBPointon mentioned this pull request Mar 7, 2025
Merged
@muffato muffato marked this pull request as ready for review March 7, 2025 15:03
@muffato
Copy link
Copy Markdown
Member Author

muffato commented Mar 7, 2025

CI is now passing (will need to update the requirement rules) and this is ready as far as I can tell

@muffato muffato mentioned this pull request Mar 7, 2025
Merged
10 tasks
DLBPointon
DLBPointon approved these changes Mar 10, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@DLBPointon DLBPointon left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me, bar the requirements of course.

@muffato muffato changed the base branch from dev to first_precomp_busco March 14, 2025 11:04
@muffato muffato merged commit 0c4640a into first_precomp_busco Mar 14, 2025
1 of 4 checks passed
@muffato muffato deleted the template_upgrade branch March 14, 2025 11:05
@muffato muffato mentioned this pull request Mar 15, 2025
Merged
10 tasks
@muffato muffato added this to the v0.7.0 milestone Feb 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DLBPointon DLBPointon DLBPointon approved these changes

@tkchafin tkchafin Awaiting requested review from tkchafin

Assignees

@muffato muffato

Labels

None yet

Projects

None yet

Milestone

v0.7.0

Development

Successfully merging this pull request may close these issues.

3 participants

@muffato @DLBPointon @tkchafin