Could I train the index data once again when I have much incremental data?

[codingcai]

Summary

Hi, all:

When I build the full amount of data, I train the data first, and then pass add_ with_ ids method create the "index" for the data. After that, I will continue to receive new data, and add it again when the data meets a certain size. The question is : before adding these, could I call the train function again to incrementally train the trained 'index'? And could the train function only be trained once before building an index?

Thank you

Platform

centos7

Faiss version:

Installed from: compiled by yourself

Faiss compilation options:

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

[mdouze]

Unfortunately there is no support for retraining after incrementally adding vectors.

[codingcai]

Thank you so much @mdouze .
I have another two questions:

1） When I build the index, I use the name "IVF4096,PQ4" to build through the index_factory . In my understanding, this index will create 4096 center points, and each vector will be compressed in PQ mode. At the same time, the quantizer is flatl2 by default. However, when I use this index to train 10W objects, on the one hand, the log prompts that there are too many 10W vectors, on the other hand, it says that training set too big. What's my opinion index.type Is there anything wrong with that? Or do I have more or less training points?

---

log output
WARNING clustering 100000 points to 4096 centroids: please provide at least 159744 training points
Training level-1 quantizer
Training level-1 quantizer on 100000 vectors in 16D
Training IVF residual
Input training set too big (max size is 65536), sampling 65536 / 100000 vectors
computing residuals
training 4x256 product quantizer on 65536 vectors in 16D
Training PQ slice 0/4

---

2） After training data with "IVFPQ*" index , and I use the add_with_ids to add vectors and ids , I want to remove some invalids ids and vectors . Could I remove the specific invalid ids so that I won't search these ids and vectors ? I look for the wiki and see the "remove operation" is only supported for Flat* ?
When I use the add_with_ids , what will happen if I use the same id with different new vectors (how to solve the old vectors)? When I search the result , will the old vector return ?

Thank you.