PairTools sort failure on merged data

[dowenlab]

Hello. I am trying to run PairTools on some mammalian HiC data. I have aligned the data with bwa and have the bam files ready. The first step (in which ${a} is the bwa alignment output bam file) finishes fine:
samtools view -h ${a} | pairtools parse -c ${Chromosomes} -o ${a}.pairfile

Then, I try the following step:
pairtools sort --nproc 8 -o ${a}.pairs.sorted ${a}.pairfile

Here, I receive an error that the program "cannot write compressed block". So, I added a flag to ensure the input is decompressed:
pairtools sort --cmd-in -o ${a}.pairs.sorted ${a}.pairfile

However, now the script no longer reads the ${a}.pairfile as input and reports that " Error: Got unexpected extra argument (Mutant-Merge.bam.pairfile)".

When I used the same script for the individual replicates before merging, everything worked fine. I merged them by simply concatenating the fastq files for the two replicates together, which I do not believe should affect this step, since the alignment ran fine.

Thank you for any feedback you may have.

[golobor]

Hi,
I apologize for the delayed response. Thank you for your interest in pairtools and your detailed report!! This looks puzzling indeed!
What operating system are you using for this example?
One issue I suspect is that you may not have write access to the Unix $TMPDIR. Could you please try running the same pairtools sort command with an extra argument --tmpdir=./ ?
Anton.

[dowenlab]

Hi,
I apologize for the delayed response. Thank you for your interest in pairtools and your detailed report!! This looks puzzling indeed!
What operating system are you using for this example?
One issue I suspect is that you may not have write access to the Unix $TMPDIR. Could you please try running the same pairtools sort command with an extra argument --tmpdir=./ ?
Anton.

Thank you so much for getting back to me! This example was using Windows 10 through an Ubuntu on Windows terminal. I have resubmitted the job including the tmpdir flag and will update to let you know how it goes!

UPDATE:
Looks like the tmpdir was the issue! The sort step successfully finished in ~3 days.

Here is the command that I ran successfully:
pairtools sort --tmpdir=./ --nproc 8 -o ${a}.pairs.sorted ${a}.pairfile

[conchoecia]

I am getting the same error with pairtools parse. Error 36 : Write error : cannot write compressed block.

The command is pairtools sort --nproc 10 -o file_parsed_sorted.pairsam.gz file_parsed.pairsam.gz, pairtools version 0.3.0, on ubuntu 18.04 LTR.

The command completed successfully when I added the --tmpdir=./ option. Maybe it would be good if this flag were a default, @golobor ?

[golobor]

@conchoecia , @dowenlab sorry for a half-year long reply :) This issue came up again, so I really need to fix it. :-(
The reason I hesitated with setting --tmpdir to ./ by default is because pairtools sort may sometimes operate on a pipe, w/o explicit input or output files. In this case, the current folder can be any folder in the file system, which would make it unpredictable. One alternative could be to attempt using $TMPDIR and, if it's inaccessible, switch to ./. But I would appreciate your experience in this issue. On the system where you got the Error 36, was the /tmp folder read-only or, instead, it had limited space that got overflowed by pairtools sort?..

[conchoecia]

Hi @golobor, in my case, and what I imagine to be the case for many people, /tmp is just a small startup disk (only a few hundred GB) on our server. The working directory is a many-TB RAID disk, so plenty of space there. Looks like our home directories probably were full, and /tmp just ran out of space.

Good luck in whatever you decide!

[golobor]

Hi @conchoecia , oh I see, thank you for the info, it's very helpful!!
I'd need to think a bit about a possible solution. There doesn't seem to be a one-size-fits-all solution - $TMPDIR is close to that, but doesn't work for a few people; using ./ by default could lead to really unpredictable behavior. A couple of things I could do is (a) to check the free space in /tmp and $TMPDIR and issue a warning if it's <10-20G; (b) improve the error message such that, if sort fails, suggest to use a different --tmpdir. Do you think that'd be useful?

[conchoecia]

Hi @golobor - Just came back to make another case for using ./ by default. Just look at what most other major bioinformatics tools do, genome assemblers, bwa/minimap/samtools, bedtools... most of these software use ./ to write tmp files. Why would your tool using ./ lead to unexpected behavior? Usually when people work in a directory, they do so because they know that there is space there and they intended to use it in the first place.

[golobor]

ok, fair enough!! I'm leaning now towards using the destination folder, if specified, otherwise in ./.

…

On Wed, 28 Oct 2020 at 17:06, darrin t schultz ***@***.***> wrote: Hi @golobor <https://github.com/golobor> - Just came back to make another case for using ./ by default. Just look at what most other major bioinformatics tools do, genome assemblers, bwa/minimap/samtools, bedtools... most of these software use ./ to write tmp files. Why would your tool using ./ lead to unexpected behavior? Usually when people work in a directory, they do so because they know that there is space there and they intended to use it in the first place. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG64CU5IC3CTEYEM5FV5XLSNA6O7ANCNFSM4NKNFVDQ> .

[agalitsyna]

Thank you for the nice discussion. There is a problem with local temporary directory that I personally don't like, it's temporary files that have to be cleaned up manually if the command is forcefully terminated. If you run multiple tests, this might lead to accumulation of temporary files, which can be easily forgotten. Other ideas/thoughts? @Phlya or others?