Improve performance (EMBLmyGFF3 v3)

[Juke34]

We have a user who has used a huge GFF annotation of 3.34 GB. It took ~ 24h computation and apparently it has used more than 50 GB of memory...
We should investigate how to optimise the speedness and the memory usage.

Thread mentioning this here.

[Juke34]

maybe avoiding in the FEATURE object
self._load_qualifier_translations(Feature.DEFAULT_QUALIFIER_TRANSLATION_FILE + translation_files)
self._load_feature_translations(Feature.DEFAULT_FEATURE_TRANSLATION_FILE)
for every feature but only once at the beginning will save lot of time.

[AdamStuckert]

Hello,

Thanks for the software, it is quite useful. I really appreciate it.

I'll start this by saying I am likely an edge-case user. I'm converting an 11 GB gff3 for a poison frog with a 6.8 Gbp genome. Currently it is 42 hours in and still writing to a flat file that is 4.4 GB. Is there an explicit flag for multithreading? I'm sure I'll have to do this again, so I'm wondering if there is a way to speed up the process.

Thanks!

[Juke34]

Are you using version2?

You can try version3 which is in the branch feature/speedup but not all features are implemented. This version is multithreaded.

Otherwise you can split you files into chunks and run them in parallel using --locus_numbering_start to avoid having similar loci name.

[Juke34]

Hi @AdamStuckert, I have made an update of the version 2 to make it faster (it is on branch "speed"). It is not multi threaded but might be 10X faster. Could you give a try on your huge file?

[AdamStuckert]

Thanks very much @Juke34! I'll install it and test it out today. I'll let you know how it does in the near future.

[Juke34]

Hi @AdamStuckert, did you succeed to run the tool from the speed branch? Did you see any improvement in term of time?

[AdamStuckert]

Hi,

Sorry for the delay. I meant to test this out and then gestures vaguely at the USA. Anyway, it ran in ~24 hours this time with the same genome assembly and annotation file, which is a really dramatic improvement in speed. Thanks!

[Juke34]

Thank you for your feedback. I have seen better performance on small files but it is still ~2 times faster on your huge file.
I will merge the branch and release a version 2.1 soon.

To resume:

• Performance has been improved in version 1.2.5 with a better way to deal with the sequence
• Performance has been improved in version 2.1 with a better way to deal with translation_files (loaded only once at the beginning then using a dictionary)
• Performance are even better in version 3 where everything has been redesigned. It is faster and parallelized. But not all options from previous versions are implemented yet, only core set of parameters. It is not yet released and can be found in branch feature/speedup