fix: store data members & make faiss tests pass

[ahuber21]

Implementations should take care of storing their own data member values when saving. The is_trained logic for LeanVec had to be adapted to allow for use cases reported in ahuber21/faiss#37

[rfsaliev]

Can you please explain, why we have to serializae/deserialize empty inidices?

[ahuber21]

For the use cases in this test
https://github.com/ahuber21/faiss/blob/b709fa114afc522b3d10ffd1356df1d9a9548951/tests/test_svs.cpp#L111

which is created to validate the scenario in this issue
ahuber21/faiss#37

[rfsaliev]

Thank you @ahuber21 for the problem description
After some time of investigation ana analysis, I got understanding, that inside the implementation code in SVS side we do not need to call DynamicVamana::save(std::ostream&) anymore, instead we can use filesystem-based DynamicVamana::save(std::filesystem::path, ...) method which allows to manage serialization in a more controlled and simpler way.

[rfsaliev]

1. I would reuse existing matrix serialization mechanizm implemented in SVS

[ahuber21]

Yes, now that it has been moved to SVS we can just use that. Is it difficult to re-use what you added for index save/load?

[rfsaliev]

I would try to extract leanvec matrices from loaded svs::leanvec::LeanDataset instead of implementing custom serialization here.

[ahuber21]

In the current implementation it's possible to have a leanvec matrix (after running train) without having an index.

[rfsaliev]

So far, to make everything consistent, shouldn't we save IndexSVSVamanaImpl members as well?

[ahuber21]

Isn't there only ntotal_soft_deleted which is always 0 because we compact on save?

Yes, I guess it makes sense to add the public members too.

[rfsaliev]

According to LeavVec training, saving and loading.

• We have the LeanVecMatrices entity - let's call it TrainingInfo abstract
• We have LeanVecIndex which can be initialized (built) with or without TrainingInfo
• LeanVecIndex should be able to be saved/loaded
• We should be able to save/load TrainingInfo and build a LeanVecIndex with pre-loaded TrainingInfo

From the SVS runtime side we should implement such functionality:

1. Build a TrainingInfo instance
2. Serialize TrainingInfo instance
3. Deserialize TrainingInfo
4. Build LeanVecIndex with TrainingInfo
5. Serialize LeanvecIndex
6. Deserialize LeanVecIndex

From FAISS side, LeanVecIndex may have one of following states:

• Empty, untrained
• Empty, trained
• Non-empty, trained

As I understand, the state 'Non-empty, untrained` - is not acceptable.
To handle these states, faiss-side logic can look like:

• IndexSVSVamanaLeanVec class fields:

  IndexPtr=NULL;
  TrainigInfoPtr=NULL;
  enum state{empty_untrained, empty_trained, non_empty};
  state get_state() { return IndexPtr ? non_empty : (TrainingInfoPtr ? empty_trained : empty_untrained); }

• IndexSVSVamanaLeanVec::train(...):

  error_if(state == non_empty)
  TrainingInfoPtr = svs::runtime::build_leanvec_training(...);
  this->trained = true;

• IndexSVSVamanaLeanVec::add(...):

  error_if(!trained);
  if (IndexPtr == NULL)
    IndexPtr = svs::build_leanvec(...TrainingInfoPtr);
  else
    IndexPtr->add(...);
  trained = true;

• IndexSVSVamanaLeanVec::serialize(...):

  error_if(!trained);
  save_state(get_state());
  if (state == empty_trained)
    TrainingInforPtr->serialize()
  else
    IndexPtr->serialize()`;

• IndexSVSVamanaLeanVec::deserialize(...):

  state = load_state();
  error_if(state == empty_untrained);
  if (state == empty_trained)
    TrainingInforPtr = svs::deserialize_training_info(...);
  else
    IndexPtr = svs::deserialize_index(...)
  trained=true