Skip to content

Add blob file type in schema.proto#241

Open
mzhl1111 wants to merge 3 commits intomilvus-io:masterfrom
mzhl1111:Feature-Support-Blob-datatype-#27578
Open

Add blob file type in schema.proto#241
mzhl1111 wants to merge 3 commits intomilvus-io:masterfrom
mzhl1111:Feature-Support-Blob-datatype-#27578

Conversation

@mzhl1111
Copy link

@mzhl1111 mzhl1111 commented Jan 25, 2024

Related Issue milvus-io/milvus#27578

@sre-ci-robot
Copy link

sre-ci-robot commented Jan 25, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mzhl1111
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot
Copy link

sre-ci-robot commented Jan 25, 2024

Welcome @mzhl1111! It looks like this is your first PR to milvus-io/milvus-proto 🎉

@mergify mergify bot added the needs-dco DCO is missing in this pull request. label Jan 25, 2024
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from 187106e to 28dc941 Compare January 25, 2024 02:52
@mergify mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Jan 25, 2024
@mzhl1111 mzhl1111 mentioned this pull request Feb 10, 2024
Open
1 task
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch 2 times, most recently from f3ec612 to 3a1aefb Compare February 11, 2024 04:15
@mergify mergify bot added the ci-passed label Feb 12, 2024
xiaofan-luan
xiaofan-luan reviewed Feb 12, 2024
View reviewed changes
proto/schema.proto Outdated
oneof field {
ScalarField scalars = 3;
VectorField vectors = 4;
ExtraField extras = 5;
Copy link
Collaborator

@xiaofan-luan xiaofan-luan Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can not change the order of proto files due to compatibility issues.
blobs could be also a scalar datatype?

Copy link
Author

@mzhl1111 mzhl1111 Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

proto/schema.proto Outdated
bool is_partition_key = 13; // enable logic partitions
bool is_clustering_key = 14;

bool is_blob_saved = 15; // for blob type, mark if the blob is saved
Copy link
Collaborator

@xiaofan-luan xiaofan-luan Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this is_blob_saved mean?

Copy link
Author

@mzhl1111 mzhl1111 Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To mark if the blob file is stored in s3 or not.

@mergify mergify bot added needs-dco DCO is missing in this pull request. and removed dco-passed DCO check passed. ci-passed labels Feb 16, 2024
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from 45900bc to b0f941f Compare February 16, 2024 19:24
@mergify mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Feb 16, 2024
@mergify mergify bot added needs-dco DCO is missing in this pull request. and removed dco-passed DCO check passed. labels Feb 17, 2024
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch 2 times, most recently from 6364bcf to 5f0d8f6 Compare February 17, 2024 02:35
@mergify mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Feb 17, 2024
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from f23b5a1 to abbe5bb Compare February 17, 2024 20:44
@mergify mergify bot added the ci-passed label Feb 19, 2024
@xiaofan-luan
Copy link
Collaborator

xiaofan-luan commented Feb 19, 2024

hi @mzhl1111,

Would you mind if we can have a quick sync on this pr?
Feel free to contact me at xiaofan.luan@zilliz.com, I'd like to have a 30 minutes quick sync.

The major concern I has:

  1. we don't need special rpc to insert blob, blob is simply like a datatype of the collection and you can simply use insert and search/query to retrieve the data.
  2. there is not much we need to change on proto side expect for schema.proto.

The most important for this pr is:

  1. at proxy, the data is written directly into object storage, and milvus itself only maintain an reference link
  2. on search, if we specify output field as blob, then file link is retrieved to proxy, proxy retrieve the data directly from object storage.

With this pr, we can store image and doc chunk into S3 so we can save great amount of memory (And maybe network traffic).

For user experience, there are no specific rpc

@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from abbe5bb to 5b23feb Compare July 11, 2024 07:07
@mergify mergify bot added needs-dco DCO is missing in this pull request. and removed dco-passed DCO check passed. ci-passed labels Jul 11, 2024
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from 5b23feb to 45520d7 Compare July 11, 2024 07:13
xiaofan-luan
xiaofan-luan reviewed Jul 15, 2024
View reviewed changes
mzhl1111 added 2 commits July 22, 2024 12:34
@mzhl1111
update with new design that blob entry stores reference
133bb07 
Signed-off-by: muzhouliu <sllzhlv77@gmail.com>
@mzhl1111
update name and bucket name entry
e934264 
Signed-off-by: muzhouliu <sllzhlv77@gmail.com>
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from 4f9614f to e934264 Compare July 22, 2024 19:34
@mergify mergify bot added dco-passed DCO check passed. and removed needs-dco DCO is missing in this pull request. labels Jul 22, 2024
@mzhl1111
update method name of saving bucket AKSK
2f7de84 
Signed-off-by: muzhouliu <sllzhlv77@gmail.com>
@mzhl1111 mzhl1111 force-pushed the Feature-Support-Blob-datatype-#27578 branch from 9463551 to 2f7de84 Compare July 23, 2024 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaofan-luan xiaofan-luan xiaofan-luan left review comments

@XuanYang-cn XuanYang-cn Awaiting requested review from XuanYang-cn

@yhmo yhmo Awaiting requested review from yhmo

Assignees

No one assigned

Labels

dco-passed DCO check passed. size/S

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mzhl1111 @sre-ci-robot @xiaofan-luan