reading index from disk and removing specified ids

[ablattmann]

Summary

I am given an index which was saved using merge_ondisk along with some metadata holding information over each vector stored in the index. Based on this metadata file, I want to remove vectors with specific ids from the index before using it for nearest neighbor search.

According to these instructions I can remove vectors from the index by using the corresponding ids for certain types of indexes. However, when loading the index with faiss.read_index(indexfile.populated, faiss.IO_FLAG_ONDISK_SAME_DIR), the result is of type indexPreTransform, which leaves me a bit puzzled. Nevertheless, I can call the index.remove_ids() function with different subclasses of IDSelector. But this will always return 0, i.e. not remove any vectors from the index. When explicitly extracting the ivf index with faiss.extract_index_ivf(index) and setting directmap to e.g. DirectMap.Hashtable via index_ivf.direct_map.set_type(faiss.DirectMap.Hashtable), this behavior will not change.

Is my aim of loading the index, removing entries corresponding to certain ids in a list of vector from which the index was created and doing a nearest neighbor search with the remaining subindex possbible at all? If yes, what am I missing? If no, are there any workarounds for this? Many thanks in advance!!

Faiss version: 1.7.2
Installed from: pip

Running on:

• [ x] CPU
• GPU

Interface:

• C++
• [ x] Python

Reproduction instructions

import os
import numpy as np
import faiss


indexfile = os.path.join(index_path,'index.populated')
# load index, index will be of type IndexPreTransform
index = faiss.read_index(indexfile, faiss.IO_FLAG_ONDISK_SAME_DIR)

# create test ids 
ids = faiss.IDSelectorArray(1000,faiss.swig_ptr(np.arange(1000)))
# try removal, will return 0
index.remove_ids(ids)


# create ivf index
index_ivf = faiss.extract_index_ivf(index)
# change direct map type to Hashtable
index_ivf.set_direct_map_type(faiss.DirectMap.Hashtable)
# try removal --> will also return 0
index_ivf.remove_ids(ids)

[mdouze]

Have you tried doing this on a non-OnDisk index? What comes to mind is that the index just not contains the specified ids.