Skip to content

Deepvariant subworkflow (issue #614)#651

Draft
fa2k wants to merge 5 commits intonf-core:devfrom
fa2k:deepvariant-workflow
Draft

Deepvariant subworkflow (issue #614)#651
fa2k wants to merge 5 commits intonf-core:devfrom
fa2k:deepvariant-workflow

Conversation

@fa2k
Copy link
Contributor

@fa2k fa2k commented Dec 6, 2024 •
edited
Loading

This PR splits DeepVariant into three processes, to optimise the resource usage. (#614)

It doesn't support GPU out of the box, but I have added usage documentation for how to enable it.

PR checklist

  • This comment contains a description of changes (with reason).
  • [na] If you've fixed a bug or added code that should be tested, add tests!
  • [na] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • [na] If necessary, also make a PR on the nf-core/raredisease branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,singularity --outdir <OUTDIR>).
  • Ensure the test suite passes (nextflow run . -profile test_one_sample,singularity --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • [na] Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • [na] README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Dec 6, 2024 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit e2963a9

+| ✅ 201 tests passed       |+
#| ❔   7 tests were ignored |#
!| ❗   2 tests had warnings |!
Details

❗ Test warnings:

❔ Tests ignored:

  • files_exist - File is ignored: conf/modules.config
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-raredisease_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-raredisease_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-raredisease_logo_dark.png
  • modules_config - modules_config

✅ Tests passed:

Run details

  • nf-core/tools version 3.0.2
  • Run at 2024-12-06 12:32:25

@ramprasadn
Copy link
Collaborator

ramprasadn commented Mar 18, 2025

Hey @fa2k! How's this one coming along? I've got some time and can jump in if you're busy. 😄

@fa2k
Copy link
Contributor Author

fa2k commented Mar 18, 2025

Hi @ramprasadn it's nice that you're looking into this, but the subworkflow solution is proving to be a bit troublesome. DeepVariant made some large changes to the pipeline structure with version 1.8.0, but me and @fellen31 have updated the subworkflow so it works again. The work to integrate new versions was a bit too much, both for the subworkflow itself, and for the pipelines using it (none at the moment AFAIK), as they must update the ext.args config.

Furthermore DeepVariant has added a "fast pipeline" with the new version, which is an alternative way to optimise the resource usage (instead of the subworkflow approach). The fast pipeline is not very user friendly at the moment, but could be improved. I'm supposed to do some benchmarking of the subworkflow vs the module, but I won't have time to do it any time soon, and I didn't plan to also include the "fast pipeline".

It may be smart to wait for the next DeepVariant release, to get a feel for the amount of changes needed for the subworkflow, and whether the fast pipeline could be an alternative.

@ramprasadn
Copy link
Collaborator

ramprasadn commented Mar 19, 2025

Okay, I will just update the module to v1.8.0 so we can take advantage of the runtime reduction achieved with the small model for now. Like you said, lets wait for the for the next release to see what DeepVariant comes up with and then decide on how to move forward.

@lpantano
Copy link

lpantano commented Mar 5, 2026

@fa2k do you want to solve the conflict and see if tests pass so we can merge?

@fellen31
Copy link
Contributor

fellen31 commented Mar 6, 2026

@fa2k do you want to solve the conflict and see if tests pass so we can merge?

Looks like running each stage of DeepVariant separately was made significantly easer in v.1.10.0 (released yesterday, but see the v1.10.0-beta release notes for examples). Maybe we could update and simplify the subworkflow?

@fa2k
Copy link
Contributor Author

fa2k commented Mar 6, 2026 •
edited
Loading

(lots of wrong info deleted)

@fa2k fa2k closed this Mar 6, 2026
@fa2k
Copy link
Contributor Author

fa2k commented Mar 6, 2026

I'm sorry, I didn't know that DV was updated. I will reopen the PR

@fa2k fa2k reopened this Mar 6, 2026
@nf-core-bot
Copy link
Member

nf-core-bot commented Mar 6, 2026

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@fa2k @ramprasadn @lpantano @fellen31 @nf-core-bot