feat: BaseVectorStorage-Add-Search-Function #2058

subway-jack · 2025-03-31T06:38:51Z

Description

Describe your changes in detail (optional if the linked issue already contains a detailed description of the changes).

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have read the CONTRIBUTION guide (required)
I have linked this PR to an issue using the Development section on the right sidebar or by adding Fixes #issue-number in the PR description (required)
I have checked if any dependencies need to be added or updated in pyproject.toml and uv lock
I have updated the tests accordingly (required for a bug fix or a new feature)
I have updated the documentation if needed:
I have added examples if this is a new feature

If you are unsure about any of these, don't hesitate to ask. We are here to help!

add the search test

Add the search function test

zjrwtx

thanks @subway-jack !left some comments

zjrwtx · 2025-03-31T08:54:08Z

camel/storages/vectordb_storages/milvus.py

            field_name="id",
            datatype=DataType.VARCHAR,
-            descrition='A unique identifier for the vector',
+            descrition="A unique identifier for the vector",


Suggested change

descrition="A unique identifier for the vector",

description="A unique identifier for the vector",

zjrwtx · 2025-03-31T08:55:55Z

please run pre-commit before commit

zjrwtx · 2025-03-31T08:57:17Z

camel/storages/vectordb_storages/base.py

+        id: str,
+        payload: Dict[str, Any],
+    ) -> "VectorDBSearchResult":
+        r"""A class method to construct a `VectorDBSearchResult` instance."""


docstring may need may clear

Thanks, I'll add more details

zjrwtx · 2025-03-31T08:57:18Z

camel/storages/vectordb_storages/base.py

+        id: str,
+        payload: Dict[str, Any],
+    ) -> "VectorDBSearchResult":
+        r"""A class method to construct a `VectorDBSearchResult` instance."""


docstring may need more clear

Thanks, I'll add more details

lightaime · 2025-03-31T09:53:31Z

Thanks @subway-jack for the contributions!

@AveryYay Hi Avery. It would be great if you can help with reviewing this PR.

add the more description detail

fix the E501

AveryYay · 2025-03-31T14:57:56Z

Thanks @subway-jack for the contributions!

@AveryYay Hi Avery. It would be great if you can help with reviewing this PR.

Sure!

AveryYay · 2025-04-04T14:04:18Z

camel/storages/vectordb_storages/milvus.py

+            if isinstance(cond, dict):
+                expressions = []
+                for op, v in cond.items():
+                    if op == "$eq":


I would consider using a dictionary to do the if-else matching here

…dd-search

hesamsheikh

Thanks for the PR @subway-jack. I have left some comments.

hesamsheikh · 2025-05-21T15:30:47Z

camel/storages/vectordb_storages/base.py

+            payload_filter (Dict[str, Any]): The filter criteria used to
+                select records.For example, {"category": "fruit", "color":
+                "red"}.
+            top_top_k (int, optional): The number of top similar vectors to


typo in top_top_k

hesamsheikh · 2025-05-21T15:31:50Z

camel/storages/vectordb_storages/base.py

+    """
+
+    record: VectorRecord
+    match_info: Dict[str, Any]


Is this actually used?

hesamsheikh · 2025-05-21T15:33:26Z

camel/storages/vectordb_storages/base.py

+    @abstractmethod
+    def search(
+        self,
+        search: VectorDBSearch,


I think this is supposed to be "filter" not search

hesamsheikh · 2025-05-21T15:34:29Z

camel/storages/vectordb_storages/base.py

        pass

+    @abstractmethod
+    def search(


maybe for the sake of name clarity, it's better to rename this function to search_by_payload, to differentiate from the filter funciton

hesamsheikh · 2025-05-21T16:12:04Z

camel/storages/vectordb_storages/milvus.py

+            if index == search.top_k - 1:
+                break


could you please make sure that the SDK does not support an argument like limit or similar?

hesamsheikh · 2025-05-21T16:14:58Z

camel/storages/vectordb_storages/qdrant.py

 from datetime import datetime
 from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast

+from qdrant_client.http.models import Filter


this is already imported multiple times in the functions

hesamsheikh · 2025-05-21T16:29:13Z

camel/storages/vectordb_storages/oceanbase.py

+    def  search(
+        self,
+        search: VectorDBSearch,
+        **kwargs: Any,
+    ) -> List[VectorDBSearchResult]:


empty function

zjrwtx · 2025-05-30T10:47:10Z

hello @subway-jack ,anything blocking you?let us know if you need any help,thanks

subway-jack · 2025-05-30T14:58:38Z

hello @subway-jack ,anything blocking you?let us know if you need any help,thanks

Thank you for the gentle reminder. I’m sorry I haven’t pushed the latest changes yet—some other tasks cropped up unexpectedly. I’ll make sure to update this PR later this week.

waleedalzarooni · 2025-11-04T12:17:44Z

Hey @subway-jack

Thanks again for contributing to this PR, would love to hear about how progress has been coming along and whether we can help in any way!

subway-jack added 11 commits March 31, 2025 14:32

Update __init__.py

22ec491

Update base.py

e2657b2

Update milvus.py

2927ada

Update qdrant.py

ea4386a

Update tidb.py

0076afa

Update __init__.py

d1c5eb1

Update tidb_vector_storage.py

942a245

Add files via upload

1773bf4

Update test_tidb.py

d1688a4

add the search test

Update test_qdrant.py

96753f5

Add the search function test

Update test_milvus.py

230b01e

Add the search function test

subway-jack requested review from Wendong-Fan, Zheng-Lu and koch3092 March 31, 2025 06:38

subway-jack self-assigned this Mar 31, 2025

subway-jack linked an issue Mar 31, 2025 that may be closed by this pull request

[Feature Request] Add search to BaseVectorStorage and all classes implemented it #1977

Open

2 tasks

subway-jack changed the title ~~Base vector storage add search~~ BaseVectorStorage-Add-Search-Function Mar 31, 2025

zjrwtx reviewed Mar 31, 2025

View reviewed changes

subway-jack added 5 commits March 31, 2025 19:28

Update base.py

3d02abf

add the more description detail

Update milvus.py

2c331ed

add the more description detail

Update qdrant.pydescription

905571b

add the more description detail

Update tidb.py

b278998

add the more description detail

Update milvus_vector_storage.py

40af846

fix the E501

AveryYay reviewed Apr 4, 2025

View reviewed changes

subway-jack added 2 commits April 25, 2025 15:01

Dealing with conflict issues

5e3f17b

"using a dictionary to do the if-else matching"

14cd84a

Merge remote-tracking branch 'origin/master' into BaseVectorStorage-a…

a36f20e

…dd-search

Wendong-Fan added the Review Required PR need to be reviewed label May 18, 2025

Wendong-Fan added this to Project Camel May 18, 2025

Wendong-Fan added this to the Sprint 30 milestone May 18, 2025

Wendong-Fan requested a review from a7m-1st May 18, 2025 19:04

Wendong-Fan changed the title ~~BaseVectorStorage-Add-Search-Function~~ feat: BaseVectorStorage-Add-Search-Function May 18, 2025

hesamsheikh self-requested a review May 21, 2025 15:23

hesamsheikh reviewed May 21, 2025

View reviewed changes

zjrwtx and others added 3 commits May 30, 2025 18:47

Merge branch 'master' into BaseVectorStorage-add-search

28a52b3

Merge 28a52b3 into b834d40

2e6ad92

Auto-update Mintlify documentation

dc1e9d9

Wendong-Fan added Waiting for Update PR has been reviewed, need to be updated based on review comment and removed Review Required PR need to be reviewed labels Jun 1, 2025

	descrition="A unique identifier for the vector",
	description="A unique identifier for the vector",

feat: BaseVectorStorage-Add-Search-Function #2058

Are you sure you want to change the base?

feat: BaseVectorStorage-Add-Search-Function #2058

Conversation

subway-jack commented Mar 31, 2025

Description

Checklist

Uh oh!

zjrwtx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjrwtx commented Mar 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightaime commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AveryYay commented Mar 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hesamsheikh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjrwtx commented May 30, 2025

Uh oh!

subway-jack commented May 30, 2025

Uh oh!

waleedalzarooni commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

lightaime commented Mar 31, 2025 •

edited

Loading