How to modify threshold of SearchParametersIVF to avoid -1

[Haroldhy]

During queries with IndexIVFFlat and SearchParametersIVF, I occasionally get an index value of -1. How should I modify the threshold to address this?
thanks
`
import faiss
import numpy as np

dimension = 5
n_vectors = 1000

vectors = np.random.rand(n_vectors, dimension).astype(np.float32)

nlist = 10
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFFlat(quantizer, dimension, nlist)

index.train(vectors)
index.add(vectors)

index.nprobe = nlist

query_vector = np.random.rand(1, dimension).astype(np.float32)

distances, indices = index.search(query_vector, k=3)

specific_indices = np.array([10, 20, 30, 40, 50], dtype=np.int64)

if hasattr(faiss, 'SearchParametersIVF'):
ivf_params_array = faiss.SearchParametersIVF()
selector_array = faiss.IDSelectorArray(len(specific_indices), faiss.swig_ptr(specific_indices))
ivf_params_array.sel = selector_array
array_distances, array_indices = index.search(query_vector, k=3, params=ivf_params_array)

ivf_params_batch = faiss.SearchParametersIVF()
selector_batch = faiss.IDSelectorBatch(len(specific_indices), faiss.swig_ptr(specific_indices))
ivf_params_batch.sel = selector_batch
batch_distances, batch_indices = index.search(query_vector, k=3, params=ivf_params_batch)

bitmap = np.zeros(n_vectors, dtype=np.uint8)
for idx in specific_indices:
    bitmap[idx] = 1

marked_indices = np.where(bitmap == 1)[0]

ivf_params_bitmap = faiss.SearchParametersIVF()
selector_bitmap = faiss.IDSelectorBitmap(n_vectors, faiss.swig_ptr(bitmap))
ivf_params_bitmap.sel = selector_bitmap
bitmap_distances, bitmap_indices = index.search(query_vector, k=3, params=ivf_params_bitmap)

manual_distances = {}
for idx in specific_indices:
    dist = np.sum((query_vector[0] - vectors[idx]) ** 2)
    manual_distances[idx] = dist

sorted_indices = sorted(manual_distances.keys(), key=lambda x: manual_distances[x])

`
IDSelectorArray and IDSelectorBatch output:[[2.4531160e-01 5.5956078e-01 3.4028235e+38]], index:[[30 40 -1]]
IDSelectorBitmap output: [3.4028235e+38 3.4028235e+38 3.4028235e+38]] and [[-1 -1 -1]] (The bitmap results don't match other methods. Occasionally, strange values like 300, -1, or -1 appear.")

[mnorris11]

SearchParametersIVF by default has nprobe 1. You have 10 clusters (nlist). If you set nprobe to 10 as well in SearchParametersIVF, does this still reproduce?

[Haroldhy]

SearchParametersIVF by default has nprobe 1. You have 10 clusters (nlist). If you set nprobe to 10 as well in SearchParametersIVF, does this still reproduce?

"Yes, I also initially thought it might be this reason. I even tried setting nlist = nprobe, but the behavior remained the same. Additionally, I attempted using the non-IVF version of SearchParameters (with the index modified to brute-force search) and found the same results. Therefore, I believe this may not necessarily be an issue exclusive to IVF."

After extensive testing, I found that this issue occurs when there are insufficient similar vectors in the vector database. Therefore, I suspect there might be a distance threshold or similar setting involved.