Description
Describe the bug
When I run the latest qmotif on WGS files, org.qcmg.picard.SAMFileReaderFactory seems to keep spawning processes, and then the program fails with Too many open files. The --thread argument doesn't seem to help in this case.
To Reproduce
qmotif_cmd = "java -Xmx512G -jar /home/scai/tools/adamajava/qmotif/build/flat/qmotif.jar --threads 8 " \ f"--input-bam {BAM} " \ f"--input-bai {BAM}.bai " \ f"--log /home/scai/SangerTMM/qmotif-1.2/results_wgs/{cell_line}/{cell_line}.qmotif.log " \ "-ini /home/scai/SangerTMM/qmotif-1.2/config.ini " \ f"-output-xml /home/scai/SangerTMM/qmotif-1.2/results_wgs/{cell_line}/{cell_line}.xml " \ f"-output-bam /home/scai/SangerTMM/qmotif-1.2/results_wgs/{cell_line}/{cell_line}.telomere.bam"
Screenshots
Desktop (please complete the following information):
- OS: CentOS Linux 7 (Core)
- qmotif version: GitHub head version.
Additional context
I could run qmotif successfully on WES data. SAMFileReaderFactory also spawns many processes but it was able to finish properly.
Thank you so much for your help!
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
ChristinaXu2017 commentedon Mar 20, 2023
Based on your error message, the program died before reaching the thread step. Could I have look at your ini file? I doubt there may be too many [INCLUDES] regions.
ZhaoxiangSimonCai commentedon Mar 20, 2023
Thank you for your reply Christina. I'm using the default [INCLUDES] regions from your documentation page.
Also I tried running qmotif on the WGS data of another sample that has deeper sequencing coverage, and it actually worked well. Could it be something that's in the bam file that causes this issue?
ChristinaXu2017 commentedon Mar 20, 2023
I am also curious why you ask for so much RAM. this tool is not RAM-hungry. I just ran on my own WGS data with the same ini file. It works fine.
ChristinaXu2017 commentedon Mar 20, 2023
Your error happens during reading the BAM file with a provided index file. It is possible something wrong with the index file or the BAM file. Alternatively, make sure the file is not moving during qmotif running.
ZhaoxiangSimonCai commentedon Mar 20, 2023
Our server has enough RAM so I set it to 512G just in case. Changing to 32G does not solve the problem.
No file is moving during running qmotif.
I tried to index the bam again but it is still the same.
Is it possible to provide more details about where the error happens, and what could be wrong? From simply viewing the BAM I can't see any problem.
ChristinaXu2017 commentedon Mar 20, 2023
It seems qmotif works well with WES BAMs and some WGS BAM, but throw the exception "Too many open files" for some WGS BAMs. I just test it with a 38G WGS BAM with 4 threads, and the run was completed in less than 10 minutes. The INI file is from our document page as well.
I am also sure your problem is nothing to do with the "thread" option. your error happens before the parallel jobs are called. Because I can't test your particular BAM file, only based on the screenshot message and our code investigation, the exception says "FileNotFound", so I guess it may be caused by:
I am sorry that I really can't provide strong opinions without a real testing dataset.
ZhaoxiangSimonCai commentedon Mar 21, 2023
Thanks again. I managed to run qmotif with setting
includes_only=truein the config.I couldn't find much about this parameter on your documentation website or paper. Could you please help explain the difference if possible? I thought qmotif always only looked at the specified regions?