[Proposal] Support Cross-Modal Search in OpenSearch

[owaiskazi19]

Is your feature request related to a problem? Please describe

Overview

Currently, OpenSearch supports multi modal search to search text and image data using multimodal embedding models. This proposal suggests implementing cross-modal search capabilities in OpenSearch, enabling users to search across different data modalities (text, images, audio, video) simultaneously and effectively.

Motivation

Modern search applications require the ability to handle multiple data types and find relationships between different modalities.

Use Cases

1. E-commerce

   • Search products using both images and text descriptions
   • Find visually similar products with different textual descriptions

2. Content Management

   • Search for media assets using text descriptions
   • Find related content across different formats

3. Digital Asset Management

   • Search through images, videos, and documents simultaneously
   • Find relationships between different content types

4. Healthcare

   • Search medical images using text descriptions
   • Match patient records with relevant medical imaging

Describe the solution you'd like

The solution can leverage OpenSearch's text search and vector search capabilities to create a unified search experience across different types of content (text, image.). It can work by converting all content into both searchable text and vector embeddings, then combines the search results using a weighted scoring system to provide the most relevant results regardless of the content type.

Related component

No response

Describe alternatives you've considered

No response

Additional context

No response

[mch2]

@owaiskazi19 This sounds great but can you elaborate more on what the current gaps are within OS and next steps you'd like to see?