k-means - get the sizes of clusters and every element in a cluster

[varvaralitvinova]

I have used k-means clustering on 180K vectors, I have 20 clusters and I would like to see how many elements each cluster contains. How can I do that?

[mdouze]

np.bincount(assign) should do the trick

[Lorisyy]

np.bincount(assign) should do the trick

As far as I'm concerned, it just shows the histogram of each element in the whole 180K vector's appearing count, but not actually the sizes of clusters and every element in a cluster

[Lorisyy]

Oops, btw you can use the tools mentioned here: https://github.com/facebookresearch/faiss/wiki/Faiss-building-blocks:-clustering,-PCA,-quantization#assignment
This computes the mapping from your set of vectors to the cluster centroids after Kmeans has finished training.
:)

[dzenilee]

I'm not sure @Lorisyy's link has the answer to @varvaralitvinova's question, but for getting counts, we can do:

from collections import Counter

counter = Counter()
for c in I.reshape(-1):
    counter[c] += 1 

sorted(counter.items())

Returns:

# (cluster id, count of elements)

[(0, 55),
 (1, 31),
 (2, 16),
 (3, 76),
 (4, 42),
 (5, 33),
 (6, 50),
...

However, it'd really be nice to have a built-in method to extract the elements of each cluster easily.

[strive-yang]

I try to use two methods to get the amount of each cluster. My data is 100M * 1536D and 10 class
1️⃣ easy
Use a dictionary to record the number of occurrences of each sample. This way must to go through all the samples.

def cluster_num(I:np.ndarray,ncentroids:int)->None:
    clNum={i:0 for i in range(ncentroids)}
    for i in I.reshape(1,-1)[0]:
        clNum[i]+=1
    print(f"The num of each cluster：\n{clNum}")

Thist way will comsume 0.7367s.
The result as below:
The num of each cluster： {0: 68108, 1: 143494, 2: 146389, 3: 76646, 4: 157293, 5: 121302, 6: 36385, 7: 82355, 8: 99759, 9: 68269}

2️⃣ use np.bincount
just np.bincount(I.reshape(1,-1)[0]).
Thist way will comsume 0.0048s.