Skip to content

Use qsv/cat for merging GUNC summaries#1004

Merged
dialvarezs merged 9 commits into
nf-core:devfrom
dialvarezs:fix/stabilize-checkm-summary
Mar 29, 2026
Merged

Use qsv/cat for merging GUNC summaries#1004
dialvarezs merged 9 commits into
nf-core:devfrom
dialvarezs:fix/stabilize-checkm-summary

Conversation

@dialvarezs

@dialvarezs dialvarezs commented Mar 27, 2026

Copy link
Copy Markdown
Member

And add sorting step to make snapshots stable

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@dialvarezs dialvarezs marked this pull request as draft March 27, 2026 21:04
@dialvarezs dialvarezs changed the title Stabilize gunc_summary.tsv snapshot Use qsv/cat for merging gunc summaries Mar 27, 2026
@dialvarezs dialvarezs changed the title Use qsv/cat for merging gunc summaries Use qsv/cat for merging GUNC summaries Mar 27, 2026
@dialvarezs dialvarezs marked this pull request as ready for review March 28, 2026 16:05
@dialvarezs dialvarezs marked this pull request as draft March 28, 2026 16:20
@dialvarezs dialvarezs marked this pull request as ready for review March 28, 2026 22:12

@jfy133 jfy133 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's fine, although I still wonder if the overhead of a whole process just for merging/sorting TSV tables is worth it computationally...

why did the sort in the collectFile not work properly again?

Maybe at least customise the base.config config to something really low (e.g. 1GB RAM, 1-2 CPUs or something)

@dialvarezs

dialvarezs commented Mar 29, 2026

Copy link
Copy Markdown
Member Author

TBH when it didn't work even when sorting by hash, I stopped trying with collectFile().
Personally I like it more this way, because it's cleaner, more maintainable, and consistent with the rest of the code. And a single light process per run is a small price to pay. I agree about setting less resources for it, going to do that.

@dialvarezs dialvarezs merged commit 957c273 into nf-core:dev Mar 29, 2026
23 checks passed
@dialvarezs dialvarezs mentioned this pull request Mar 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants