Problem
The current implementation sets K using a heuristic: K = int(sqrt(num_vectors / 2)*10) if your training data includes at least 100,000 records. This means that there is an accuracy shift that kicks in at exactly 100,000 records. Before this point, the system will retrieve the nearest neighbor from all 100,000 points but after this point we will be searching from 500 records.
While the elbow method offers a more statistically grounded way to choose K by evaluating clustering performance across several K values, it is computationally intensive.
Solution
Implementing the elbow method would require running KMeans multiple times and analyzing metrics like SSE or silhouette scores. Given our real-time and batch constraints, we avoid this due to diminishing accuracy gains (often logarithmic) versus increased computational cost (often linear to exponential with larger datasets).
However, since this value can be derived during the batch ingestion, when performance is not a priority, we could add it there. :)
Additional information
No response
👨👧👦 Contributing
Problem
The current implementation sets K using a heuristic: K = int(sqrt(num_vectors / 2)*10) if your training data includes at least 100,000 records. This means that there is an accuracy shift that kicks in at exactly 100,000 records. Before this point, the system will retrieve the nearest neighbor from all 100,000 points but after this point we will be searching from 500 records.
While the elbow method offers a more statistically grounded way to choose K by evaluating clustering performance across several K values, it is computationally intensive.
Solution
Implementing the elbow method would require running KMeans multiple times and analyzing metrics like SSE or silhouette scores. Given our real-time and batch constraints, we avoid this due to diminishing accuracy gains (often logarithmic) versus increased computational cost (often linear to exponential with larger datasets).
However, since this value can be derived during the batch ingestion, when performance is not a priority, we could add it there. :)
Additional information
No response
👨👧👦 Contributing