FAISS_ASSERT(allocs_.count(device) == 0) failured!

[iclgg]

I am using Faiss-GPU, and my project occasionally encounters the error Faiss assertion 'allocs_.count(device) == 0' failed in virtual void faiss::gpu::StandardGpuResourcesImpl::initializeForDevice(int) at /xx/xx/faiss/gpu/StandardGpuResources.cpp:347 after running normally for some time. This causes the service to stop.
Below is the code in my project for using GPU search:

def __init__(self,*args):
    ...
    self.gpu_res = faiss.StandardGpuResources() # use a single GPU
    ...
def search_gpu(self, embeddings):
    num = embeddings.shape[0]
    self.rwlock.acquire_read() 
    try:
        if self.index.ntotal == 0:
            return np.full((num, self.k), -1, dtype=np.float32), np.full((num, self.k), -1, dtype=np.int64)
        gpu_index = faiss.index_cpu_to_gpu(self.gpu_res, 0, self.index)
        D, I = gpu_index.search(embeddings, self.k)
    finally:
        self.rwlock.release_read() 
    return D, I

This service is a web service. Could the error be caused by concurrency issues leading to a failure in StandardGpuResources()? Please give me some possible reasons so I can investigate, such as CUDA version, Faiss version, or other factors?

faiss-gpu version: v1.7.4

[asadoughi]

It reads like you are creating a new gpu_index for every request which would be problematic. Consider making the lifecycle of gpu_index as the same as the service, not the same as the lifecycle of the request.

[iclgg]

Thank you! I’ll give it a try.