This sample demonstrates how to build a real-time voice assistant with Retrieval-Augmented Generation (RAG) capabilities using Azure AI Voice Live API and Azure AI Search. It showcases a complete end-to-end solution where users can have natural voice conversations with an AI agent that retrieves and references information from your own documents and knowledge base. With Voice Live API, developers can easily voice-enable any agent built with the Azure AI Foundry Agent Service.
- Real-Time Voice Conversations: Bi-directional speech-to-speech interaction powered by Voice live
- RAG Integration: Seamless integration with Azure AI Search for document retrieval and grounded responses
- Full-Stack Architecture: React/TypeScript frontend with FastAPI backend, deployable to Azure Container Apps
- Azure AI Foundry Agents: Voice-enable agents built with the Azure AI Foundry Agent Service
- Production-Ready Deployment: Complete
azdtemplate for one-command deployment to Azure
- azd, used to deploy all Azure resources and assets used in this sample.
- PowerShell Core pwsh if using Windows
- Python 3.10
- An Azure Subscription with Contributor permissions
- Sign in to Azure with Azure CLI
- VS Code installed with the Jupyter notebook extension enabled
┌─────────────────────────────────────┐ ┌──────────────────┐ ┌─────────────────────┐
│ Azure Container App │ │ Azure Voice │ │ Azure AI Search │
│ │ │ Live API │ │ │
│ ┌─────────────────────────────────┐│ │ │ │ │
│ │ Frontend Client ││ │ ┌──────────────┐ │ │ ┌─────────────────┐ │
│ │ (React/TypeScript) ││ │ │ GPT-4o │ │ │ │ Search Index │ │
│ │ ││ │ │ Realtime │ │ │ │ & Retrieval │ │
│ │ ┌─────────────────────────────┐ ││ │ │ Processing │ │ │ └─────────────────┘ │
│ │ │ Microphone Input │ ││ │ └──────────────┘ │ │ │
│ │ │ Speaker Output │ ││ │ │ │ ┌─────────────────┐ │
│ │ │ Tool Call UI Display │ ││ │ │ │ │ Vector Store │ │
│ │ └─────────────────────────────┘ ││ │ │ │ │ & Documents │ │
│ └─────────────────┬───────────────┘│ │ │ │ └─────────────────┘ │
│ │ WebSocket │ │ │ │ │
│ ┌─────────────────▼───────────────┐│ │ │ │ │
│ │ Backend API ││ │ │ │ │ WebSocket
│ │ (FastAPI) ││◄──►│ │◄──►│ │
│ │ ││ │ │ │ │
│ │ ┌─────────────────────────────┐ ││ │ │ │ │
│ │ │ WebSocket Handler │ ││ │ │ │ │
│ │ │ Audio Streaming │ ││ │ │ │ │
│ │ │ Tool Execution │ ││ │ │ │ │
│ │ └─────────────────────────────┘ ││ │ │ │ │
│ └─────────────────────────────────┘│ │ │ │ │
└─────────────────────────────────────┘ └──────────────────┘ └─────────────────────┘
This sample uses azd and a bicep template to deploy all Azure resources:
Create the infrastructure
# Login to Azure (if not already logged in)
az login
# Initialize the project (if running for the first time)
azd init
# Deploy infrastructure and application to Azure
azd upAfter the application has been successfully deployed you will see a URL printed to the console. Navigate to that URL to interact with the app in your browser.
Ask questions like What is the due date of my bill? and What are the benefits of my credit card?
azd down --purge