Support newer versions of Milvus

[codingjaguar]

Is your feature request related to a problem? Please describe.
It seems the SDK supports up to Milvus 2.2 version. There are a few changes in Milvus 2.4 and 2.5, and we greatly appreciate if those can be supported in C# SDK!

Describe the solution you'd like

• connection parameter: uri and token as recommended; host, port, username, password are kept for backward compatability.
• introducing BM25 Function for full text search, see python and java SDK for example: https://milvus.io/docs/full-text-search.md

Describe alternatives you've considered
N/A

Additional context
N/A

[codingjaguar]

Regarding connection parameter:

Until this is fixed, the best workaround is to set port to -1 if there is no port (for example Zilliz Cloud serverless clusters doesn't have port in the URI). For regular self-hosted Milvus and Zilliz Cloud dedicated, you can set host as the url, and port as the port (e.g. 19530).

[roji]

Hey @weianweigan, any chance you're planning to do a bit of work to bring the SDK up to date?

[ahusseini]

what's going on with this SDK?

[roji]

This SDK is currently lacking a stable owner. While I did the initial work of putting it together and maintaining it for a while, I'm unable to continue owning it long-term; someone from either Milvus or the community will need to step up to do periodic maintenance, bringing it up to date with new features, etc.

I'm more than happy to continue advising/helping out and review the occasional PR, but I can't be the sole person owning this SDK.

[xiaofan-luan]

This SDK is currently lacking a stable owner. While I did the initial work of putting it together and maintaining it for a while, I'm unable to continue owning it long-term; someone from either Milvus or the community will need to step up to do periodic maintenance, bringing it up to date with new features, etc.

I'm more than happy to continue advising/helping out and review the occasional PR, but I can't be the sole person owning this SDK.

unfortunately, none of the core team in milvus is familiar in C#.
is there someone willing to take the ownership of this repo. We'd like to offer help like monthly sync about current progress of Milvus progress and help on setting up some ci/cd

[roji]

Copying list of features since Milvus 2.2 which @yhmo posted in #102 (comment):

The following list is the key features that each Milvus SDK needs to be implemented on the client-side(Maybe I missed something). You can take a look at the interfaces of Milvus C++ SDK: https://github.com/milvus-io/milvus-sdk-cpp/blob/2.5/src/include/milvus/MilvusClient.h. Most of the v2.5 features have been implemented in the C++ SDK(almost one hundred interfaces).

v2.3

Features

• SCANN index
• GPU_IVF_FLAT/GPU_IVF_PQ index
• Support QueryIterator/SearchIterator
• Cosine metric
• Upsert()
• DescribeAlias()/ListAliases()
• Range Search
• Array field
• Dynamic field
• Delete by filter expression

Other changes

• Remove index types: IVF_HNSW/RHNSW_FLAT/RHNSW_SQ/RHNSW_PQ/ANNOY
• Remove metrics: TANIMOTO/Superstructure/Substructure
• Deprecated Time Travel

v2.4

Features

• Add new index types: DISKANN/AUTOINDEX/GPU_BRUTE_FORCE/GPU_CAGRA/TRIE/STL_SORT/INVERTED/SPARSE_INVERTED_INDEX/SPARSE_WAND
• Add new metric types: COSINE
• SparseVector field
• Multi-Embedding fields in one collection
• Float16Vector/BFloat16Vector field
• HybridSearch()
• AlterCollectionField()
• AlterIndexProperties()/DropIndexProperties()
• Partial load
• Allow upsert to auto-id collection
• RBAC enhancements, new interfaces

Improvement

• Cache collection schema to reduce DescribeCollection call
• Support search with grouping by field
• Retry machinery for RPC call
• UseDatabase()
• Row-based insert/upsert
• QuertResults/SearchResults to return row-based results

Other changes

• Remove CalcDistance interface, not supported by milvus 2.4
• Remove Flush call in GetCollectionStats/GetPartitionStats/CreateIndex, milvus 2.4 doesn't allow calling Flush constantly
• CreateIndex() internally calls DescribeIndex() to check index state instead of GetIndexState()
• Remove client-side checks for index parameters and search parameters, let milvus server validate

v2.5

Features

• Add new index types: Bitmap
• HybridSearch supports group by
• Clustering compaction
• Nullable field
• Default value of field
• Text match
• Full text search(BM25 function)
• RunAnalyzer()
• DropCollectionProperties()/DropCollectionFieldProperties()

v2.6

Features

• Add new index types: MINHASH_LSH/IVF_RABITQ/NGRAM
• Int8Vector
• AddCollectionField()
• Rerank functions for Search/HybridSearch
• Struct field
• Geometry field
• Timestamptz field

Optional Tools

• BulkWriter
• BulkImport by RESTful API
• Client pool

[roji]

Whoever starts working on bringing the SDK up to date - I'd advise creating issues in this repo based on the list above and then start tackling them.

[verdie-g]

@roji I think it would be very valuable to annotate this list with your knowledge of the client.

To give an example:

Deprecated Time Travel

Maybe this feature (whatever that is) was not implemented in the .NET client so it can already be striked.

What do you think about adding a [ ] in front of all of the items and add your comment for each?

[roji]

I honestly don't have much to add - almost of all of these are new things that appeared after I worked on the driver, so I don't think that would be very valuable.

Someone basically needs to start working on these :)

[verdie-g]

Copying with checkboxes to track the progress.

v2.3

Features

• SCANN index Add SCANN index #113
• GPU_IVF_FLAT/GPU_IVF_PQ index
• Support QueryIterator/SearchIterator Implement QueryWithIterator/SearchWithIterator #115
• Cosine metric
• Upsert()
• DescribeAlias()/ListAliases() Implement describe and list alias #112
• Range Search
• Array field Fix support for double, arrays, and binary vectors #107
• Dynamic field
• Delete by filter expression

Other changes

• Remove index types: IVF_HNSW/RHNSW_FLAT/RHNSW_SQ/RHNSW_PQ/ANNOY
• Remove metrics: TANIMOTO/Superstructure/Substructure
• Deprecated Time Travel

v2.4

Features

• Add new index types: DISKANN/AUTOINDEX/GPU_BRUTE_FORCE/GPU_CAGRA
• Add new index types: TRIE/STL_SORT/INVERTED Add index TRIE, STL_SORT, INVERTED #117
• Add new metric types: COSINE
• Add new index types: SPARSE_INVERTED_INDEX/SPARSE_WAND Support sparse float vector #127
• SparseVector field Support sparse float vector #127
• Multi-Embedding fields in one collection Support float16 vector #114
• Float16Vector field Support float16 vector #114
• BFloat16Vector field Support the BFloat16 vector data type #76
• HybridSearch() Implement hybrid search #125
• AlterCollectionField()
• AlterIndexProperties()/DropIndexProperties()
• Partial load
• Allow upsert to auto-id collection
• RBAC enhancements, new interfaces

Improvement

• Cache collection schema to reduce DescribeCollection call
• Support search with grouping by field Implement missing search parameters #126
• Retry machinery for RPC call
• UseDatabase()
• Row-based insert/upsert
• QueryResults/SearchResults to return row-based results

Other changes

• Remove CalcDistance interface, not supported by milvus 2.4
• Remove Flush call in GetCollectionStats/GetPartitionStats/CreateIndex, milvus 2.4 doesn't allow calling Flush constantly
• CreateIndex() internally calls DescribeIndex() to check index state instead of GetIndexState()
• Remove client-side checks for index parameters and search parameters, let milvus server validate

v2.5

Features

• Add new index types: Bitmap
• HybridSearch supports group by Implement hybrid search #125
• Clustering compaction
• Nullable field Support default and nullable fields #118
• Default value of field Support default and nullable fields #118
• Text match Implement BM25 #134
• Full text search(BM25 function) Implement BM25 #134
• RunAnalyzer()
• DropCollectionProperties()/DropCollectionFieldProperties()

v2.6

Features

• Add new index types: MINHASH_LSH/IVF_RABITQ/NGRAM
• Int8Vector
• AddCollectionField()
• Rerank functions for Search
• Rerank functions for HybridSearch Implement hybrid search #125
• Struct field
• Geometry field
• Timestamptz field

Optional Tools

• BulkWriter
• BulkImport by RESTful API
• Client pool

[yhmo]

@roji @verdie-g
Feel free to let me know if you have any questions about the features. Once I finish the CPP SDK work, I will join you to upgrade the C # SDK.