This sample highlights an approach to building a copilot for E-commerce, bringing in a multimodal experience to enable a personal concierge-like experience for shoppers. It combines the power of Azure OpenAI Service and Azure AI Search for indexing and retrieving multimodal content. The solution not only helps users find and recommend products based on their shopping needs but also leverages the power of multimodality. Whether they are preparing for an event, looking for matching styles, or even using an image to find the perfect match, the copilot has them covered.
- Use Case
- Features
- Architecture
- Getting Started
- Code of Conduct
- Responsible AI Guidelines
- Dataset License
- License
This solution enables users to explore retail products such as clothing and accessories by interacting with the bot in natural language. Its multimodal capabilities ensure seamless experiences; for example, users can upload an image of their suit, and the bot will suggest matching shoes or accessories.
Imagine getting ready for a party and finding a complete outfit—simply by conversing with the bot. Whether you're searching for a suit, shoes, or complementary accessories, the bot provides relevant and tailored suggestions.
To showcase the solution’s capabilities, a pre-recorded voiceover demonstrates its functionality, ranging from simple queries to complex multimodal interactions. Watch the demo video below and follow along with the Demo Script.
retail_demo.mp4
The repository includes a complete end-to-end solution, comprising:
- A frontend application for seamless user interaction
- Backend microservices to handle core functionalities
- A synthetically generated dataset of product catalog items to quickly set up and test the solution
- A generic ingestion service that enhances chunks with metadata (e.g., product descriptions using images) to drastically improve search results
The solution includes the following key components:
-
Ingestion Service: The ingestion service includes various enhancements to ensure that when raw content is ingested, there is minimum loss of information. Additional metadata is added to the chunks to the index to make search results more relevant.
-
Enhanced User Interaction: This includes a frontend with multimodal capabilities for users to interact with the bot and an implementation of a queuing layer in the backend. This allows users to send multiple questions to the copilot, and the copilot can produce multiple responses to a user query, making the overall experience more engaging.
-
Core Microservices and Skills: This includes the Orchestrator, which executes various skills to best address the user query. The core microservices handle different aspects of the solution, such as session management, data processing, runtime configuration, and orchestration. Specialized skills provide specific capabilities, such as AI search and image describer.
-
Testing and Evaluation: This includes the ability to simulate conversations with the copilot, run certain end-to-end tests on demand, and an evaluation tool to help perform end-to-end evaluation of the copilot.
Detailed architecture for the eCommerce Copilot can be found here
To set up and start using this project, follow our Getting Started Guide. It provides step-by-step instructions for both Azure resources and local environments.
This project has adopted the Microsoft Open Source Code of Conduct.
Resources:
- Microsoft Open Source Code of Conduct
- Microsoft Code of Conduct FAQ
- Contact [email protected] with questions or concerns
For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
This project follows below responsible AI guidelines and best practices, please review them before using this project:
- Microsoft Responsible AI Guidelines
- Responsible AI practices for Azure OpenAI models
- Safety evaluations transparency notes
This dataset is released under the Community Data License Agreement – Permissive, Version 2.0 - CDLA, see the LICENSE-DATA file.
This project is licensed under the MIT License. See the LICENSE file for details.