Computing KMeans for datasets that don't fit in memory

[nicolas-dufour]

Hi,
Looking at the doc, i see that when training faiss.Kmeans, we need to give the full dataset as an array.
However can it be possible to give it in batches for when we cannot load the full dataset in memory ? Or provide a pytorch Dataloader?

Otherwise what's the recommended way to handle very big datasets?

Thanks!

[mdouze]

If you have a very large dataset it probably means that you also have many centroids. In that case you probably want to use a distributed k-means implementation, see
https://github.com/facebookresearch/faiss/blob/main/benchs/distributed_ondisk/distributed_kmeans.py

[nicolas-dufour]

Thanks for the suggestion @mdouze ! Do you have some idea of the order of data/centroids it's better to use the distributed method? Myself i was thinking of clustering around 10M-20M embeddings with 50K to 100K centroids.