Fix test profile memory configuration for xengsort#4
Merged
tylergross97 merged 1 commit intoFeb 20, 2026
Merged
Conversation
Resolved out-of-memory errors in XENGSORT_CLASSIFY by: 1. Made xengsort index size configurable via ext.args - Updated xengsort_index.nf to accept custom args - Default remains -n 4500000000 -k 25 for production use 2. Optimized test profile for small test data - Reduced index size to -n 100000000 (22x smaller) for test data - Increased XENGSORT_CLASSIFY memory from 4GB to 8GB - Smaller index loads faster and uses less RAM This allows the test profile to run successfully with tiny test files while maintaining production settings for real data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The test profile was failing with out-of-memory errors during the
XENGSORT_CLASSIFYstep:All three XENGSORT_CLASSIFY tasks were killed after multiple retry attempts, despite tiny test files (only a few reads each).
Root Cause
The xengsort index size was hardcoded to
-n 4500000000(4.5 billion k-mers), which requires significant RAM to load into memory during classification. Even with tiny test FASTQ files, the index size remained large and exceeded the allocated 4GB memory limit.Solution
This PR implements two key improvements:
1. Made index size configurable
modules/xengsort_index.nfto accept custom arguments viatask.ext.args-n 4500000000 -k 25for production use2. Optimized test profile
-n 100000000(22x smaller, appropriate for test data)Changes
modules/xengsort_index.nf: Made index parameters configurable viaext.argsnextflow.config: Updated test profile with:-n 100000000)Testing
The test profile should now complete successfully:
nextflow run tylergross97/nextflow_xengsort -profile test -revision seqera-ai/20260220-181546-fix-test-memory-configImpact
ext.args