Skip to content

Subworkflow update#132

Closed
DLBPointon wants to merge 72 commits intodevfrom
subworkflow-update
Closed

Subworkflow update#132
DLBPointon wants to merge 72 commits intodevfrom
subworkflow-update

Conversation

@DLBPointon
Copy link
Contributor

@DLBPointon DLBPointon commented Feb 6, 2026

#130 - FOFN or FILE input
#129 - Update to use Jim's Module

@prototaxites - Can you have a look at the implementation of your hic mapping subworkflow into curationpretext please?

I've replaced the current method and am using the below to test (otherwise it will die as i havn't replaced the pretext mapper bits yet:

nextflow run /lustre/scratch124/tol/teams/tola/users/dp24/curationpretext/main.nf -profile test,sanger,singularity --skip_tracks ALL

The test data is in the above dir. Essentially, processes are occasionally getting past here:

    ch_hic_cram_meta_mod = ch_hic_cram
        .transpose()
        .map { meta, cram -> [ meta + [ cramfile: cram ], cram ]}

But no further.

Input cram channel to the subworkflow is:

[[id:CurationPretextTest], [/lustre/scratch124/tol/teams/tola/users/dp24/curationpretext/TreeValTinyData/genomic_data/hic-arima/SUBSET-1000.cram, /lustre/scratch124/tol/teams/tola/users/dp24/curationpretext/TreeValTinyData/genomic_data/hic-arima/SUBSET-2000.cram]]

Pipeline will freeze like:

executor >  lsf (4)
[-        ] SANGER_TOL_CURATIONPRETEXT:CURATIONPRETEXT:GUNZIP                          -
[07/419b31] SAN…ATIONPRETEXT:CURATIONPRETEXT:GAWK_UPPER_SEQUENCE (CurationPretextTest) [100%] 1 of 1 ✔
[df/14942d] SAN…ETEXT:CURATIONPRETEXT:SAMTOOLS_FAIDX (CurationPretextTest_UPPER.fasta) [100%] 1 of 1 ✔
[-        ] SANGER_TOL_CURATIONPRETEXT:CURATIONPRETEXT:ALIGN_CRAM:SAMTOOLS_INDEX       -
[-        ] SAN…TOL_CURATIONPRETEXT:CURATIONPRETEXT:ALIGN_CRAM:CRAMALIGN_GENCRAMCHUNKS -
[-        ] SANGER_TOL_CURATIONPRETEXT:CURATIONPRETEXT:ALIGN_CRAM:SAMTOOLS_SPLITHEADER -
[04/3679f1] SAN…TIONPRETEXT:ALIGN_CRAM:BWAMEM2_INDEX (CurationPretextTest_UPPER.fasta) [100%] 1 of 1 ✔
[-        ] SAN…L_CURATIONPRETEXT:CURATIONPRETEXT:ALIGN_CRAM:CRAMALIGN_BWAMEM2ALIGNHIC -
[dd/4981dd] SAN…AMTOOLS_MERGE_MARKDUP:SAMTOOLS_FAIDX (CurationPretextTest_UPPER.fasta) [100%] 1 of 1 ✔
[-        ] SAN…URATIONPRETEXT:ALIGN_CRAM:BAM_SAMTOOLS_MERGE_MARKDUP:SAMTOOLS_MERGEDUP -

Then dies with no error:

ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-[sanger-tol/curationpretext] Pipeline completed with errors-

.nextflow.log shows nothing, presumably I'm passing in a dogey channel, but everything looks fine to me.

Thanks

Edit: This can wait till Monday so feel free to ignore!

@DLBPointon DLBPointon changed the base branch from main to dev February 6, 2026 22:44
@DLBPointon DLBPointon marked this pull request as draft February 6, 2026 22:44
@sanger-tol sanger-tol deleted a comment from github-actions bot Feb 6, 2026
@github-actions
Copy link

github-actions bot commented Feb 6, 2026

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 7d281fc

+| ✅ 185 tests passed       |+
#| ❔  32 tests were ignored |#
#| ❔   1 tests had warnings |#
!| ❗   3 tests had warnings |!
Details

❗ Test warnings:

  • nextflow_config - Config manifest.version should end in dev: 1.6.0
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • schema_lint - Input mimetype is missing or empty

❔ Tests ignored:

  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • files_exist - File is ignored: assets/multiqc_config.yml
  • files_exist - File is ignored: assets/nf-core-curationpretext_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-curationpretext_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-curationpretext_logo_dark.png
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_exist - File is ignored: CODE_OF_CONDUCT.md
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File does not exist: CODE_OF_CONDUCT.md
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/feature_request.yml
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File does not exist: assets/nf-core-curationpretext_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-curationpretext_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-curationpretext_logo_dark.png
  • files_unchanged - File ignored due to lint config: docs/README.md
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/curationpretext/curationpretext/.github/workflows/awstest.yml
  • multiqc_config - multiqc_config

❔ Tests fixed:

✅ Tests passed:

Run details

  • nf-core/tools version 3.5.2
  • Run at 2026-02-27 11:44:15

@prototaxites
Copy link
Contributor

Hi @DLBPointon, will take a look in depth tomorrow but just to double check - do the cram and the assembly channels share a meta? This is needed in order to ensure the joins within work.

@DLBPointon
Copy link
Contributor Author

DLBPointon commented Feb 10, 2026

No problem, thanks.

And it does indeed now but still an issue, this is from inside the subworkflow.

image

@prototaxites
Copy link
Contributor

This is very strange, I don't understand what is happening. It looks like nothing goes through that .transpose() block, but I cannot at all see why that would be.

@prototaxites
Copy link
Contributor

I found the problem. .transpose() only works on channel objects, but your function fn_get_validated_channel does not actually return a channel, just a tuple.

I fixed it so that it returns a channel instead, and now mapping runs. You could alteratively revert that change and do:

channel.of(fn_get_validated_channel(...))

if you prefer.

@DLBPointon
Copy link
Contributor Author

I found the problem. .transpose() only works on channel objects, but your function fn_get_validated_channel does not actually return a channel, just a tuple.

I fixed it so that it returns a channel instead, and now mapping runs. You could alteratively revert that change and do:

channel.of(fn_get_validated_channel(...))

if you prefer.

Absolute legend, thanks a ton Jim. Now onto adding pairs_create_contact_maps.

Copy link
Contributor

@prototaxites prototaxites left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of implementation thoughts!

Copy link
Contributor

@prototaxites prototaxites left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, one last comment about documentation

@DLBPointon
Copy link
Contributor Author

Somhow managed to poison the PR

No matter what, I can't get it into a usable state. Once the branch is switched to, then it is poisoned and useless. Mind boggling issue.

Instead I spent saturday morning re-writting the PR before the family got up, otherwise I would have thought about it all weekend.

new PR is: #136

@DLBPointon DLBPointon closed this Mar 2, 2026
@github-project-automation github-project-automation bot moved this from In progress to Done in WRAP team board Mar 2, 2026
@DLBPointon DLBPointon moved this from Done to No Longer Planned in WRAP team board Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

3 participants