Seq-hash to load input data

[ShriramK]

galaxy_blast/tools/blast_rbh/blast_rbh.py

Line 270 in d31e21d

# TODO - seq-hash based to avoid loading everything into RAM?

[peterjc]

Are you interested in working on this? What do you think I meant by this comment? Are you finding the current implementation a problem?

[ShriramK]

Yes. It makes sense to not load large data sets/files into RAM.
I'm trying to figure out the Sequential hashing part.

[peterjc]

To date with mainly single organism BLAST databases this functionality has not been a problem for me, and no one has reported an issue with the tool running out of memory. So this is a low priority, and does come with risks and downsides.

There are a lot of possible hashing functions that could be used, from the Python built-in hash, CRC64, MD5, SHA256, or even the sequence specific SEGUID (available in Biopython). Some of these are going to be slow, perhaps enough to be a problem on large datasets.

Some of these will have collisions - so should we check for that explicitly (e.g. with another round of disk IO on output, again slow)? CRC64 in particular would be a bad choice, see the 44CAAD88706CC153 example in https://github.com/biopython/biopython/blob/biopython-171/Tests/test_SeqUtils.py#L40