Need 500 GB of RAm for 2000 assemblies?

[tseemann]

I am trying to run ska2 with 2000 assembled genoemes, but ska merge ends up taking 4 days and then runs out of RAM at ~500 GB (the server has 512 GB RAM).

Is there a way around this problem?

eg. mash lets me provide a FOFN of sketchs to mash dist, it does not requore a merge first. But i suspect your merge is more complex/Venn-like than just cat-ting sketehces

Is ska build -o merged *.fna -j 64 the better way to do this? Will it use less RAM?

Thanks for any help!

[rderelle]

The memory consumption is expected to increase with the number of genomes but I didn't expect that much. I remember having built skf files from 700-800 Mtb genomes and the RAM consumption stayed below 40 GB. Perhaps are you analysing species with a large open pan-genome (?).

Two ways around this problem using the current version of ska2:

• [best] partition your set of genomes into MLST, strains, lineages ... and then build n independent skf files
• [alternative] built skf file using successive chunks of the dataset, with a ska weed step in between each build (see Fig. 5 in the SKA2 paper). It is very effective at lowering RAM consumption but SNPs are lost, particularly those yet unseen -> best for data storage but less so for further addition of new genomes.

Hope this helps.