Skip to content

Fix test profile memory configuration for xengsort#4

Merged
tylergross97 merged 1 commit into
mainfrom
seqera-ai/20260220-181546-fix-test-memory-config
Feb 20, 2026
Merged

Fix test profile memory configuration for xengsort#4
tylergross97 merged 1 commit into
mainfrom
seqera-ai/20260220-181546-fix-test-memory-config

Conversation

@tylergross97

Copy link
Copy Markdown
Owner

Problem

The test profile was failing with out-of-memory errors during the XENGSORT_CLASSIFY step:

ERROR ~ Essential container in task exited - OutOfMemoryError: Container killed due to memory usage
Exit code: 137

All three XENGSORT_CLASSIFY tasks were killed after multiple retry attempts, despite tiny test files (only a few reads each).

Root Cause

The xengsort index size was hardcoded to -n 4500000000 (4.5 billion k-mers), which requires significant RAM to load into memory during classification. Even with tiny test FASTQ files, the index size remained large and exceeded the allocated 4GB memory limit.

Solution

This PR implements two key improvements:

1. Made index size configurable

  • Updated modules/xengsort_index.nf to accept custom arguments via task.ext.args
  • Default remains -n 4500000000 -k 25 for production use
  • Allows profiles to override index parameters

2. Optimized test profile

  • Reduced index size: -n 100000000 (22x smaller, appropriate for test data)
  • Increased XENGSORT_CLASSIFY memory: 4GB → 8GB
  • Smaller index loads faster and requires less RAM

Changes

  • modules/xengsort_index.nf: Made index parameters configurable via ext.args
  • nextflow.config: Updated test profile with:
    • Smaller index size for test data (-n 100000000)
    • Increased memory for XENGSORT_CLASSIFY (8GB)

Testing

The test profile should now complete successfully:

nextflow run tylergross97/nextflow_xengsort -profile test -revision seqera-ai/20260220-181546-fix-test-memory-config

Impact

  • ✅ Test profile now works with tiny test files
  • ✅ Production workflows unaffected (use default 4.5B k-mer index)
  • ✅ Custom profiles can override index size as needed via ext.args

Resolved out-of-memory errors in XENGSORT_CLASSIFY by:

1. Made xengsort index size configurable via ext.args
   - Updated xengsort_index.nf to accept custom args
   - Default remains -n 4500000000 -k 25 for production use

2. Optimized test profile for small test data
   - Reduced index size to -n 100000000 (22x smaller) for test data
   - Increased XENGSORT_CLASSIFY memory from 4GB to 8GB
   - Smaller index loads faster and uses less RAM

This allows the test profile to run successfully with tiny test files
while maintaining production settings for real data.
@tylergross97 tylergross97 added the seqera-ai Pull request created by Seqera AI label Feb 20, 2026
@tylergross97 tylergross97 merged commit a55a653 into main Feb 20, 2026
4 checks passed
@tylergross97 tylergross97 deleted the seqera-ai/20260220-181546-fix-test-memory-config branch February 20, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

seqera-ai Pull request created by Seqera AI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant