[Spinoff from https://github.com//issues/572]
Modern BLAS/GEMM libraries are insanely optimized -- see Claude's explanation:
GEMM is the single most optimized piece of numerical code on the planet — it's been hand-tuned for thirty years. The kernels use register blocking, L1/L2/L3 cache blocking, FMA instructions, and AVX-512 (or on your Threadripper 3990X, AVX2 — Zen 2 doesn't do AVX-512, but it still does 256-bit FMAs at very high throughput). It also parallelizes across all cores automatically.
Let's just build a pluggable vector math impl for Lucene (compile the right dotProduct.so) using this? We can test in benchies here vs Panama, gcc autovectorize, etc.
One issue might be that Lucene doesn't bulk up enough -- these GEMM kernel impls work on "tiles" (multiple doc vectors scored against query vector in a subset of dimensions or so), so we should see sizable gains if we make our vector math calls more bulky. When checking all neighbors of the node we just pulled out of the HNSW exploration queue, do we bulk score all of its neighbors (unvisited) vectors? If we rerank by higher precision vector score, is that all bulk? When aggregating all vectors during indexing to compute quantization params, is that bulk...
[Spinoff from https://github.com//issues/572]
Modern BLAS/GEMM libraries are insanely optimized -- see Claude's explanation:
Let's just build a pluggable vector math impl for Lucene (compile the right
dotProduct.so) using this? We can test in benchies here vs Panama, gcc autovectorize, etc.One issue might be that Lucene doesn't bulk up enough -- these GEMM kernel impls work on "tiles" (multiple doc vectors scored against query vector in a subset of dimensions or so), so we should see sizable gains if we make our vector math calls more bulky. When checking all neighbors of the node we just pulled out of the HNSW exploration queue, do we bulk score all of its neighbors (unvisited) vectors? If we rerank by higher precision vector score, is that all bulk? When aggregating all vectors during indexing to compute quantization params, is that bulk...