When performing a multi-index merge, why should the coarse quantizers be the same ?

[guoxinghai]

Summary

I trained the two indexes separately, but when I merged the two indexes using merge_from() I got the following error:
RuntimeError: Error in virtual void faiss::IndexIVF::check_compatible_for_merge(const faiss::Index&) const at /root/workspace/faiss/IndexIVF.cpp:1210: Error: 'v == v2' failed: coarse quantizers should be the same
The code I use is as follows:

import faiss
import numpy as np



dim = 256
nlist = 1448
nprobe = 2
k = 4

dnum = 55600
qnum = 640

# load feature
feature1 = np.array(np.random.randn(dnum, dim), dtype=np.float32)
feature2 = np.array(np.random.randn(dnum, dim), dtype=np.float32)

# index
quantizer1 = faiss.IndexFlatL2(dim)
index1 = faiss.IndexIVFFlat(quantizer1, dim, nlist)
index1.nprobe = nprobe

quantizer2 = faiss.IndexFlatL2(dim)
index2 = faiss.IndexIVFFlat(quantizer2, dim, nlist)
index2.nprobe = nprobe


# train
index1.train(feature1)
index2.train(feature2)

# add
ids = np.arange(0, dnum, dtype=np.int64)
index1.add_with_ids(feature1, ids)
index2.add_with_ids(feature2, ids)

# merge
index1.merge_from(index2, index1.ntotal)

In addition, I modified the index2 quantizer before the merge_from() (after train() and add()), and it worked. Is there any problem with this.

index2.quantizer = copy.deepcopy(index1.quantizer)
index1.merge_from(index2, index1.ntotal)

Platform

OS: ubuntu 20.04

Faiss version: V1.8.0

Running on:

• [√ ] CPU
• GPU

Interface:

• C++
• [√ ] Python

Reproduction instructions

[guoxinghai]

Through the experiment, I found that the index performance obtained by the merge after re-specifying the quantizer for index2 would significantly decrease. Will quantizers also participate in the 'train' and 'add' processes. Is there another way to merge two indices that use different quantizers?

[guoxinghai]

I used python to train two indexes in multiple processes (created by the same quantizer), and a Segmentation fault (core dumped) occurred during the merge. Is there a method for distributed training multiple indexes and merging them into one index