Skip to content

Deterministic tx2gene sample selection in quant_tximport_summarizedexperiment - #1875

Closed
pinin4fjords wants to merge 1 commit into
devfrom
tx2gene-deterministic-resume
Closed

Deterministic tx2gene sample selection in quant_tximport_summarizedexperiment#1875
pinin4fjords wants to merge 1 commit into
devfrom
tx2gene-deterministic-resume

Conversation

@pinin4fjords

@pinin4fjords pinin4fjords commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

Bumps the quant_tximport_summarizedexperiment subworkflow to pull in nf-core/modules#12166 and nf-core/modules#12168.

In QUANT_TXIMPORT_SUMMARIZEDEXPERIMENT, the quant results fed to CUSTOM_TX2GENE were selected with .first(). tx2gene only needs a single sample's quant files (it reads them to find which GTF attribute holds the transcript IDs; all samples share a transcriptome), so picking one sample is correct - but .first() picks whichever per-sample quant arrives first, and queue-channel arrival order equals task-completion order, which is non-deterministic across runs.

That made CUSTOM_TX2GENE's input non-deterministic: a different sample's quant directory was staged on different runs, the task hash changed, and -resume re-ran tx2gene even when nothing changed. Since TXIMETA_TXIMPORT and the SummarizedExperiment builds depend on CUSTOM_TX2GENE.out.tx2gene, the cache miss cascaded through the subworkflow.

The subworkflow now selects the sample deterministically, with a guard for the empty-channel case:

ch_tx2gene_quants = quant_results
    .toSortedList { a, b -> a[1].name <=> b[1].name }
    .filter { sorted -> sorted.size() > 0 }
    .map { sorted -> [ [:], sorted.first()[1] ] }

The chosen sample is arbitrary biologically (tx2gene output is identical for any sample sharing the transcriptome), so pipeline outputs and snapshots are unchanged - this only makes the selection reproducible so -resume works. The .filter preserves the original no-op-on-empty behaviour for runs that invoke the subworkflow with no quant results (e.g. the unused alignment path when only pseudo-alignment runs).

Changes

PR checklist

  • This comment contains a description of changes (with reason).
  • nf-core subworkflows lint quant_tximport_summarizedexperiment passes.
  • Verified against tests/kallisto.nf.test --skip_alignment; no pipeline output or snapshot change.
  • CHANGELOG.md is updated.
  • Documentation not affected.

🤖 Generated with Claude Code

pinin4fjords added a commit that referenced this pull request Jun 25, 2026
@pinin4fjords @claude
docs: add CHANGELOG entry for PR #1875
817cce6 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 0da4e0a

+| ✅ 215 tests passed       |+
#| ❔  19 tests were ignored |#
!| ❗   7 tests had warnings |!
Details

❗ Test warnings:

  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

❔ Tests ignored:

  • files_exist - File is ignored: conf/modules.config
  • files_exist - File is ignored: conf/containers_conda_lock_files_amd64.config
  • files_exist - File is ignored: conf/containers_conda_lock_files_arm64.config
  • files_exist - File is ignored: conf/containers_docker_amd64.config
  • files_exist - File is ignored: conf/containers_docker_arm64.config
  • files_exist - File is ignored: conf/containers_singularity_https_amd64.config
  • files_exist - File is ignored: conf/containers_singularity_https_arm64.config
  • files_exist - File is ignored: conf/containers_singularity_oras_amd64.config
  • files_exist - File is ignored: conf/containers_singularity_oras_arm64.config
  • nextflow_config - Config default ignored: params.ribo_database_manifest
  • nf_test_content - nf_test_content
  • files_unchanged - File ignored due to lint config: assets/email_template.html
  • files_unchanged - File ignored due to lint config: assets/email_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-rnaseq_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-rnaseq_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
  • actions_nf_test - actions_nf_test
  • modules_config - modules_config
  • container_configs - container_configs

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.2
  • Run at 2026-07-15 16:07:40

@pinin4fjords

pinin4fjords commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

Added an empty-quant_results guard (.filter { sorted -> sorted.size() > 0 }): toSortedList always emits an empty list for an empty channel, so sorted.first() aborted runs where the subworkflow is invoked with no quant results (the unused alignment path when only pseudo-alignment runs). This restores the prior .first() no-op-on-empty behaviour while keeping deterministic selection. Verified against tests/kallisto.nf.test --skip_alignment - passes with snapshots unchanged.

Expected failure: the nf-core subworkflow lint check will fail until nf-core/modules#12168 merges, because the local subworkflow copy now carries this guard ahead of the pinned upstream sha. Once #12168 is in, I'll re-sync the git_sha and the lint check will pass.

@pinin4fjords

pinin4fjords commented Jun 26, 2026

Copy link
Copy Markdown
Member Author

Re-synced to nf-core/modules#12168 (now merged) and bumped the git_sha. The local subworkflow copy is byte-identical to upstream at the merge commit, so the subworkflow lint check now passes too. Full CI re-running.

@pinin4fjords pinin4fjords mentioned this pull request Jul 2, 2026
Closed
@pinin4fjords pinin4fjords mentioned this pull request Jul 15, 2026
Merged
3 tasks
@pinin4fjords @claude
fix: deterministic tx2gene sample selection in quant_tximport_summari…
3588a45 
…zedexperiment

CUSTOM_TX2GENE was fed whichever per-sample quant arrived first, which varies
between runs since queue-channel arrival order follows task-completion order.
That made tx2gene's input non-deterministic, changing its task hash and
breaking -resume for it and everything downstream (TXIMETA_TXIMPORT, the
SummarizedExperiment builds).

The sample is now chosen by sorting on the staged results name, with a guard
for the empty-channel case (toSortedList still emits an empty list, unlike
the previous .first() which was a no-op on empty input).

Bumps quant_tximport_summarizedexperiment to nf-core/modules#12168, which
carries this fix upstream.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pinin4fjords
pinin4fjords force-pushed the tx2gene-deterministic-resume branch from 0da4e0a to 3588a45 Compare July 24, 2026 15:04
@pinin4fjords

pinin4fjords commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

Closing this, #1885 already bumped quant_tximport_summarizedexperiment to a later revision (nf-core/modules#12377) that includes this exact fix (originally nf-core/modules#12166) plus additional sorting for TXIMETA_TXIMPORT/CUSTOM_RSEMMERGECOUNTS. Diffed the subworkflow file against current dev and it's byte-identical to what's here, so there's nothing left for this PR to add.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pinin4fjords