Data storage and usage in Multimodal data scenarios

[littleDrew]

Here in Multimodal data scenarios, like involve image, audio, vedio case, i am thinking whether users will store these kinds of multimodal data into deeplake, or just storage directly in disk like S3?

Hoping have a disscuss for this to enhance understanding in multimodal data scenarios.

Possible Usages of Multimodal data

Before consider how Multimodal data will be stored, here we should consider how will these kinds of data will be used later, here i thought may exist this kinds of usage:

• Convert images, text, audio, etc., into embedding vectors to facilitate downstream retrieval and analysis. Here may store embedding vectors into deeplake.
• Tagging images and videos.
• feed image or video to model, here include read image or frame of video and convert to normalized Tensor and as input for model.

How will or should Multimodal data be stored ?

Currently, here in mind, it seems save metadata and path of Multimodal data is more common, like following table data:

id	image_path	text (string)	embedding (float[768])	label (string)	meta (struct)
1	dir/0001.JPG	one cat	[0.1, 0.2,...]	cat	{"size": "1024x168", ...}
2	dir/0002.JPG	a dog	[0.3, 0.5,...]	dog	{"size": "1024x512", ...}

• Thus here i want to know is there some case that user will tends to store Multimodal data like image or video into deeplake directly instead of remote disk like S3?

Hoping you share you ideas and insight @activesoull @artgish @zaaram @tatevikh @khustup @dgaloop

[khustup2]

Hey @littleDrew,
Storing the data directly in deeplake will help you with simplicity on accessing data directly from deeplake. It also can have performance benefits because deeplake will parallelize data fetching and image decompression under the hood.

If you need to access those images outside of deeplake, then you can have column type Link[Image] and pass the links to deeplake. It will be treated as image under the hood and deeplake will download links for you - https://docs.deeplake.ai/latest/api/types/#deeplake.types.Link

We don't support Audio and Video data yet, although you can use Bytes type to store raw bytes if needed. We will add video type support soon. This will give you fast random access to the video frames.

Hope this helps.

[littleDrew]

Thanks for your reply, you refered there are two ways to store Multimodal data:

• 1st way: directly store image/video in deeplake
• 2nd way: save link/path of image/video in deeplake but actual data in remote disk like s3.

May be you have better understanding as for actual user's Multimodal data scenarios, could you show some public actual case like which user will select 1st way, that store image/video in deeplake directly : ) @khustup2

[khustup2]

Thanks for your reply, you refered there are two ways to store Multimodal data:

• 1st way: directly store image/video in deeplake
• 2nd way: save link/path of image/video in deeplake but actual data in remote disk like s3.

May be you have better understanding as for actual user's Multimodal data scenarios, could you show some public actual case like which user will select 1st way, that store image/video in deeplake directly : ) @khustup2

@littleDrew I don't have public examples showing each use case, but the main usage of links are the following:

• You already have your data(image, video) in the storage and want to avoid data duplication
• You use the same data outside of deeplake and want to have external access to it.

For all other scenarios we suggest to keep your data directly in deeplake, because the data access with that setup will be the fastest.