FAISS GPU Baseline script issue with MSSPACEV dataset #289
Description
I'm trying to repro the faiss GPU baseline results with MSSPACEV dataset. I'm following the instructions using the [FAISS T3 Baseline] (https://github.com/harsha-simhadri/big-ann-benchmarks/tree/main/neurips21/track3_baseline_faiss).
Built FAISS using source from the main branch.
Then executed the faiss baseline script and running into the below issue:
/big-ann-benchmarks/neurips21/track3_baseline_faiss$ python3 gpu_baseline_faiss.py --dataset msspacev-1M --indexkey IVF65536,SQ8 --train_on_gpu --build --quantize
r_on_gpu_add --add_splits 30 --search --searchparams nprobe={1,4,16,64,256} --parallel_mode 3 --quantizer_on_gpu_search
nb processors 64
model name : Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
Dataset MSSPACEV1B in dimension 100, with distance euclidean, search_type knn, size: Q 29316 B 1000000
build index, key= IVF65536,SQ8
Build-time number of threads: 64
Update add-time parameters
setting maxtrain to 3276800
getting first 3276800 dataset vectors for training
data/MSSPACEV1B/spacev1b_base.i8bin
train, size (1000000, 100)
add a training index on GPU
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 100D
WARNING clustering 1000000 points to 65536 centroids: please provide at least 2555904 training points
Clustering 1000000 points in 100D to 65536 clusters, redo 1 times, 10 iterations
Preprocessing in 0.28 s
Iteration 9 (6.14 s, search 5.43 s): objective=6.03294e+09 imbalance=1.691 nsplit=0
Training IVF residual
Input training set too big (max size is 100000), sampling 100000 / 1000000 vectors
Total train time 50.826 s
adding
============== SPLIT 0/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 0:33333 / 1000000 [0.223 s, RSS 3757956 kiB]
Traceback (most recent call last):
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 591, in <module>
main()
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 543, in main
index = build_index(args, ds)
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 210, in build_index
index.add_core(
File "/mnt/ssd_volume/mumba1/lib/python3.10/site-packages/faiss-1.8.0-py3.10.egg/faiss/swigfaiss.py", line 6956, in add_core
return _swigfaiss.IndexIVFScalarQuantizer_add_core(self, n, x, xids, precomputed_idx, inverted_list_context)
TypeError: Wrong number or type of arguments for overloaded function 'IndexIVFScalarQuantizer_add_core'.
Possible C/C++ prototypes are:
faiss::IndexIVFScalarQuantizer::add_core(faiss::idx_t,float const *,faiss::idx_t const *,faiss::idx_t const *,void *)
faiss::IndexIVFScalarQuantizer::add_core(faiss::idx_t,float const *,faiss::idx_t const *,faiss::idx_t const *)this appeared to be an issue with the faiss swig interface not being able to properly cast from the numpy datatype passed to the faiss library. Hence, tried to manually cast them to satisfy the compiler.
+ temp = xblock.astype('f4')
+ temp1 = assign.astype('i8')
index.add_core(
len(xblock),
- faiss.swig_ptr(xblock),
+ faiss.swig_ptr(temp),
None,
- faiss.swig_ptr(assign)
+ faiss.swig_ptr(temp1)
)
The compiler errors are gone, but running into the below runtime exception.
/big-ann-benchmarks/neurips21/track3_baseline_faiss$ (base) cs21resch14001@gpuserver:/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss$ vi gpu_bapython3 gpu_baseline_faiss.py --dataset msspacev-1M --indexkey IVF65536,SQ8 --train_on_gpu --build --quantizer_on_gpu_add --add_splits 30 --search --searchparams nprobe={1,4,16,64,256} --parallel_mode 3 --quantizer_on_gpu_search
nb processors 64
model name : Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
Dataset MSSPACEV1B in dimension 100, with distance euclidean, search_type knn, size: Q 29316 B 1000000
build index, key= IVF65536,SQ8
Build-time number of threads: 64
Update add-time parameters
setting maxtrain to 3276800
getting first 3276800 dataset vectors for training
data/MSSPACEV1B/spacev1b_base.i8bin
train, size (1000000, 100)
add a training index on GPU
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 100D
WARNING clustering 1000000 points to 65536 centroids: please provide at least 2555904 training points
Clustering 1000000 points in 100D to 65536 clusters, redo 1 times, 10 iterations
Preprocessing in 0.29 s
Iteration 9 (6.83 s, search 5.99 s): objective=6.03294e+09 imbalance=1.691 nsplit=0
Training IVF residual
Input training set too big (max size is 100000), sampling 100000 / 1000000 vectors
Total train time 55.841 s
adding
============== SPLIT 0/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 0:33333 / 1000000 [0.272 s, RSS 3744696 kiB]
============== SPLIT 1/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 33333:66666 / 1000000 [0.332 s, RSS 3759812 kiB]
============== SPLIT 2/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 66666:100000 / 1000000 [0.421 s, RSS 3787152 kiB]
============== SPLIT 3/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 100000:133333 / 1000000 [0.477 s, RSS 3787208 kiB]
============== SPLIT 4/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 133333:166666 / 1000000 [0.533 s, RSS 3792648 kiB]
============== SPLIT 5/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 166666:200000 / 1000000 [0.589 s, RSS 3796876 kiB]
============== SPLIT 6/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 200000:233333 / 1000000 [0.622 s, RSS 3801216 kiB]
============== SPLIT 7/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 233333:266666 / 1000000 [0.672 s, RSS 3824248 kiB]
============== SPLIT 8/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 266666:300000 / 1000000 [0.747 s, RSS 3827172 kiB]
============== SPLIT 9/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 300000:333333 / 1000000 [0.817 s, RSS 3830356 kiB]
============== SPLIT 10/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 333333:366666 / 1000000 [0.857 s, RSS 3836412 kiB]
============== SPLIT 11/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 366666:400000 / 1000000 [0.895 s, RSS 3842488 kiB]
============== SPLIT 12/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 400000:433333 / 1000000 [0.962 s, RSS 3846732 kiB]
============== SPLIT 13/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 433333:466666 / 1000000 [1.010 s, RSS 3851480 kiB]
============== SPLIT 14/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 466666:500000 / 1000000 [1.055 s, RSS 3858196 kiB]
============== SPLIT 15/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 500000:533333 / 1000000 [1.102 s, RSS 3861880 kiB]
============== SPLIT 16/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 533333:566666 / 1000000 [1.166 s, RSS 3883036 kiB]
============== SPLIT 17/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 566666:600000 / 1000000 [1.195 s, RSS 3886160 kiB]
============== SPLIT 18/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 600000:633333 / 1000000 [1.225 s, RSS 3891904 kiB]
============== SPLIT 19/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 633333:666666 / 1000000 [1.254 s, RSS 3895884 kiB]
============== SPLIT 20/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 666666:700000 / 1000000 [1.292 s, RSS 3902764 kiB]
============== SPLIT 21/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 700000:733333 / 1000000 [1.499 s, RSS 3920764 kiB]
============== SPLIT 22/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 733333:766666 / 1000000 [1.539 s, RSS 3924992 kiB]
============== SPLIT 23/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 766666:800000 / 1000000 [1.682 s, RSS 3947948 kiB]
============== SPLIT 24/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 800000:833333 / 1000000 [1.754 s, RSS 3949752 kiB]
============== SPLIT 25/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 833333:866666 / 1000000 [1.795 s, RSS 3954252 kiB]
============== SPLIT 26/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 866666:900000 / 1000000 [1.828 s, RSS 3959776 kiB]
============== SPLIT 27/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 900000:933333 / 1000000 [1.877 s, RSS 3966388 kiB]
============== SPLIT 28/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 933333:966666 / 1000000 [1.921 s, RSS 3969824 kiB]
============== SPLIT 29/30
data/MSSPACEV1B/spacev1b_base.i8bin
adding 966666:1000000 / 1000000 [1.973 s, RSS 3974576 kiB]
add in 2.092 s
imbalance_factor= 1.690308313088
current RSS: 3439468544
precomputed tables size: 0
Search threads: 64
setting IVF parallel mode to 3
Optimize for intersection @ 10
Running evaluation on 5 searchparams
parameters inter@ 10 time(ms/q) nb distances %quantization #runs
nprobe=1 Traceback (most recent call last):
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 591, in <module>
main()
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 588, in main
run_experiments_searchparams(ds, index, args)
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 434, in run_experiments_searchparams
eval_setting_knn(
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 248, in eval_setting_knn
D, I = index.search(xq, k)
File "/mnt/ssd_volume/big-ann-benchmarks/neurips21/track3_baseline_faiss/gpu_baseline_faiss.py", line 336, in search
self.index_ivf.search_preassigned(
TypeError: handle_Index.<locals>.replacement_search_preassigned() takes 5 positional arguments but 9 were given
FATAL: exception not rethrown
FATAL: exception not rethrown
Aborted (core dumped)