Introduce MinIO Object Store event based dataprep and retriever using Milvus and LanceDB

[dilverse]

Description

MinIO is a high performance Object Storage solution which is fully S3 compatible and using which its easier to build highly scalable applications. In this PR adding support for the following

• In Dataprep storing all the uploaded documents directly to MinIO
• Modularize the dataprep process
• Once the document is uploaded to MinIO bucket an event notification is sent to the dataprep service to do the chunking and store the chunked metatdata as msgpack into MinIO Bucket
• once the msgpack chunked metadata is stored in MInIO bucket another notification is send to do the embeddings process and store the chunks, metadata and embeddings to to vector database like Milvus and LanceDB.
• Also adding support for MinIO LanceDB based retriever

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

[mkbhanda]

@lvliang-intel would you kindly review.

[mkbhanda]

Looks good to me. @chensuyue do we need any tests?

[mkbhanda]

would be good to change one of these to a file path/url. What is the name of a file when it is path? id == name?

[chensuyue]

Yes, we need some one test for each microservice.
test_retrievers_minio_lancedb_langchain.sh
test_dataprep_minio_lancedb_langchain.sh
test_dataprep_minio_milvus_langchain.sh

[chensuyue]

Please refer to https://github.com/opea-project/GenAIComps/tree/main/tests/dataprep

[eero-t]

Below are few minor comments on the docs and Python code.

As to Dockerfiles, see: opea-project/GenAIExamples#225

[lvliang-intel]

@dilverse,
Please update comps/dataprep/minio/milvus/langchain/Dockerfile, comps/dataprep/minio/lancedb/langchain/Dockerfile and comps/retrievers/minio/lancedb/langchain/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

[dilverse]

@dilverse, Please update comps/dataprep/minio/milvus/langchain/Dockerfile, comps/dataprep/minio/lancedb/langchain/Dockerfile and comps/retrievers/minio/lancedb/langchain/Dockerfile to .github/workflows/docker/compose/dataprep-compose.yaml. The yaml is used for release images build.

@lvliang-intel Updated the github workflow

[xiguiw]

@dilverse
GenAIComps are refactored in v1.2.
It does not create a micro-service for each DB anymore.
Instead, only one micro-service is needed.
It simplifies the integration of DB.

Please refer to latest code change.

[xiguiw]

@dilverse

Remind
Are you still working on this?

[dilverse]

@dilverse

Remind Are you still working on this?

Sounds good I will take a look at it and update the PR @xiguiw

[xiguiw]

@dilverse

There is conflict, please help to resolve the conflicts.