Merge pull request #124 from yulin-li/yulin/voice-live-update

update voice live agent sample for FDP projects

Author: mwhalin

samples/agent-catalog/msft-agent-samples/foundry-agent-service-sdk/voice-live-agent/README.md

@@ -1,18 +1,18 @@
 # Voice Live Agent
 
-This sample showcases how to voice-enable any agents built with Azure AI Foundry Agent Service, utilizing Azure AI Voice Live API.  
+This sample showcases how to voice-enable any agents built with Azure AI Foundry Agent Service, utilizing Azure AI Voice Live API.
 
-**IMPORTANT NOTE:** Starter templates, instructions, code samples and resources in this msft-agent-samples file (“samples”) are designed to assist in accelerating development of agents for specific scenarios. It is important that you review all provided resources and carefully test Agent behavior in the context of your use case: ([Learn More](https://learn.microsoft.com/en-us/legal/cognitive-services/agents/transparency-note?context=%2Fazure%2Fai-services%2Fagents%2Fcontext%2Fcontext)). 
+**IMPORTANT NOTE:** Starter templates, instructions, code samples and resources in this msft-agent-samples file (“samples”) are designed to assist in accelerating development of agents for specific scenarios. It is important that you review all provided resources and carefully test Agent behavior in the context of your use case: ([Learn More](https://learn.microsoft.com/en-us/legal/cognitive-services/agents/transparency-note?context=%2Fazure%2Fai-services%2Fagents%2Fcontext%2Fcontext)).
 
-Certain Agent offerings may be subject to legal and regulatory requirements, may require licenses, or may not be suitable for all industries, scenarios, or use cases. By using any sample, you are acknowledging that Agents or other output created using that sample are solely your responsibility, and that you will comply with all applicable laws, regulations, and relevant safety standards, terms of service, and codes of conduct.  
+Certain Agent offerings may be subject to legal and regulatory requirements, may require licenses, or may not be suitable for all industries, scenarios, or use cases. By using any sample, you are acknowledging that Agents or other output created using that sample are solely your responsibility, and that you will comply with all applicable laws, regulations, and relevant safety standards, terms of service, and codes of conduct.
 
 ## Use cases
 
-Voice-enabled agents are high in demand, now more than ever. Voice agents are agents that users can interact with naturally and conversationally using just their voice. From an end-user perspective, voice is becoming the preferred mode of interaction, as it enables speed, accessibility, and multitasking.  
+Voice-enabled agents are high in demand, now more than ever. Voice agents are agents that users can interact with naturally and conversationally using just their voice. From an end-user perspective, voice is becoming the preferred mode of interaction, as it enables speed, accessibility, and multitasking.
 
 We see increasing demand across several key use-cases including:
 
-**Customer service** – think about getting support from your favorite department store, your bank, your travel agency, or even your government;  
+**Customer service** – think about getting support from your favorite department store, your bank, your travel agency, or even your government;
 
 **Automotive** – think about in-car assistants with hands-free interaction;
 
@@ -26,28 +26,28 @@ The system consists of:
 
 - An AI Agent created with Azure AI Agent Service. You can create an agent using any of the templates provided in agent-catalog/azure-ai-agent-service-blueprints at main · microsoft/agent-catalog
 
-- An Azure Voice Live API request. You can set up your Voice Live API request following the instructions in this document and code sample.  
+- An Azure Voice Live API request. You can set up your Voice Live API request following the instructions in this document and code sample.
 
 ```text
-+-----------------+                          
-|   User Query    |                          
-| (Speech Input)  |                          
-+-------+---------+                          
-        |                                   
-        v                                   
-+---------------------+           invokes          +----------------------------+ 
-|  Voice Live Agent   | ------------------------>  |       Azure AI Agent       | 
-|                     | <------------------------  |   + Knowledge + Actions    | 
-+---------------------+           results          +----------------------------+ 
-        |                                   
-        v                                   
-+-------------------+            
-|   Agent Response  |            
-|  (Speech Output)  |            
-+-------------------+             
++-----------------+
+|   User Query    |
+| (Speech Input)  |
++-------+---------+
+        |
+        v
++---------------------+           invokes          +----------------------------+
+|  Voice Live Agent   | ------------------------>  |       Azure AI Agent       |
+|                     | <------------------------  |   + Knowledge + Actions    |
++---------------------+           results          +----------------------------+
+        |
+        v
++-------------------+
+|   Agent Response  |
+|  (Speech Output)  |
++-------------------+
 ```
 
-## Voice Live API introduction  
+## Voice Live API introduction
 
 Azure AI Voice Live API (preview) is an innovative, unified single API that enables streaming interactions with the foundation model of your choice, for both speech input and output. It includes advanced features such as customizable speech recognition, diverse text-to-speech options, brand voices, avatars, audio enhancement, among other functionalities. With Voice Live API, you can add real-time speech interaction capabilities to any agent built with the Azure AI Agent Service.
 
@@ -61,35 +61,35 @@ A live demo (<https://aka.ms/voice-agent/demo>) is also available to experience
 
 ### Prerequisites
 
-**Set up an agent**. Follow the templates provided in agent-catalog/azure-ai-agent-service-blueprints at main · microsoft/agent-catalog to create your agent.
+**Set up an agent**. Follow [the templates](../) to create an agent using the Azure AI Agent Service.
 
 **Resource and authentication**. An Azure AI Foundry resource is required to access the Voice Live API. To learn how to create an Azure AI Foundry resource, please see: <https://learn.microsoft.com/azure/ai-services/multi-service-resource>.
 
 Note: The resource must be in the `eastus2` or `swedencentral` regions at this time. Other regions are not supported.
 
-	@@ -31,25 +87,24 @@ We support two authentication methods for the Voice Live API:
-
 For the recommended keyless authentication with Microsoft Entra ID, you need to:
 
-- Assign the `Azure AI User` role to your user account or a managed identity. You can assign roles in the Azure portal under **Access control (IAM)** 
-
-**Add role assignment**.
+- Assign the `Azure AI User` role to your user account or a managed identity. You can assign roles in the Azure portal under **Access control (IAM)** > **Add role assignment**.
 - Generate a token using the Azure CLI or Azure SDKs. The token must be generated with the `https://ai.azure.com/.default` scope.
 - Use the token in the `Authorization` header of the WebSocket connection request, with the format `Bearer <token>`.
 
 ## Set Agent Info
 
 You are supposed to specify the agent info in the WebSocket endpoint URL.
 
-| Parameter            | Description                                                                                                                                                                                                                   |
-| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `agent-project-name` | The Azure AI project name which the agent belongs to.                                                                                                                                                                         |
-| `agent-id`           | The ID of the agent to use.                                                                                                                                                                                                   |
+| Parameter            | Description                                                                                                                                                                                                              |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `agent-project-name` | The Azure AI project name which the agent belongs to.                                                                                                                                                                    |
+| `agent-id`           | The ID of the agent to use.                                                                                                                                                                                              |
 | `agent-access-token` | The Entra access token to access the agent. Make sure the identity has access to Azure AI Project, You can grant the built-in role `Azure AI User` to the identity. The scope should be `https://ai.azure.com/.default`. |
 
 > Note: The token must be generated with the `https://ai.azure.com/.default` scope. e.g., `az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv`.
-A sample endpoint is `wss://<custom-domain>.cognitiveservices.azure.com/voice-agent/realtime?api-version=2025-05-01-preview&agent-project-name=<agent-project-name>&agent-id=<agent-id>&agent-access-token=<access-token>`.
+A sample endpoint is `wss://<your-ai-foundry-resource-name>.cognitiveservices.azure.com/voice-live/realtime?api-version=2025-05-01-preview&agent-project-name=<agent-project-name>&agent-id=<agent-id>&agent-access-token=<access-token>`.
 
 ## Interact with the Voice Live API
 
-Refer to the [full documentation of Voice Live API](https://learn.microsoft.com/azure/ai-services/speech-service/voice-live) for more details on how to interact with the Voice Live API.
+Refer to the [full documentation of Voice Live API](https://learn.microsoft.com/azure/ai-services/<placeholder>) for more details on how to interact with the Voice Live API.
+
+## Getting started
+
+Follow the instructions [here](./samples/react/README.md) to get started with this sample.

samples/agent-catalog/msft-agent-samples/foundry-agent-service-sdk/voice-live-agent/samples/react/package-lock.json

@@ -184,12 +184,16 @@ const readme = `
         - The endpoint can be the regional endpoint (e.g., \`https://<region>.api.cognitive.microsoft.com/\`) or a custom domain endpoint (e.g., \`https://<custom-domain>.cognitiveservices.azure.com/\`).
         - The resource must be in the \`eastus2\` or \`swedencentral\` region. Other regions are not supported.
 
+    2. **(Optional) Set the Agent**
+        - Set the project name and agent ID to connect to a specific agent.
+        - Entra ID auth is required for agent mode, use \`az account get-access-token --resource https://ai.azure.com --query accessToken -o tsv\` to get the token.
+
     2. **Select noise suppression or echo cancellation**
         - Enable noise suppression and/or echo cancellation to improve audio quality.
 
     3. **Select the Turn Detection**
         - Choose the desired turn detection method. The default is \`Server VAD\`, which uses server-side voice activity detection.
-        - The \`Server SD (reduced false alarms)\` option is also available for better performance.
+        - The \`Azure Semantic VAD\` option is also available for better performance.
 
     4. **Select the Voice**
        - Choose the desired voice from the list.
@@ -410,9 +414,9 @@ const ChatInterface = () => {
 
   // Add mode state and agent fields
   const [mode, setMode] = useState<"model" | "agent">("model");
-  const [agentConnectionString, setAgentConnectionString] = useState("");
+  const [agentProjectName, setAgentProjectName] = useState("");
   const [agentId, setAgentId] = useState("");
-  const [agentAccessToken, setAgentAccessToken] = useState("");
+  // const [agentAccessToken, setAgentAccessToken] = useState("");
   const [agents, setAgents] = useState<{ id: string; name: string }[]>([]);
   const [isMobile, setIsMobile] = useState(false);
 
@@ -448,9 +452,9 @@ const ChatInterface = () => {
           setPredefinedScenarios(config.pre_defined_scenarios);
         }
         // Parse agent configs from /config
-        if (config.agent && config.agent.connection_string) {
-          setAgentAccessToken(config.agent.access_token);
-          setAgentConnectionString(config.agent.connection_string);
+        if (config.agent && config.agent.project_name) {
+          // setAgentAccessToken(config.agent.access_token);
+          setAgentProjectName(config.agent.project_name);
           if (Array.isArray(config.agent.agents)) {
             setAgents(config.agent.agents);
             // If only one agent, auto-select it
@@ -520,8 +524,8 @@ const ChatInterface = () => {
             ? {
               modelOrAgent: {
                 agentId,
-                agentConnectionString,
-                agentAccessToken,
+                projectName: agentProjectName,
+                agentAccessToken: entraToken,
               },
               apiVersion: "2025-05-01-preview",
             }
@@ -1174,6 +1178,7 @@ const ChatInterface = () => {
       "gpt-4.1",
       "gpt-4.1-mini",
       "gpt-4.1-nano",
+      "phi4-mini",
     ];
     return cascadedModels.includes(model);
   }
@@ -1253,24 +1258,33 @@ const ChatInterface = () => {
                   onChange={(e) => setEndpoint(e.target.value)}
                   disabled={isConnected || configLoaded}
                 />
-                {!configLoaded && (
+                {(!configLoaded && mode === "model") && (
                   <Input
                     placeholder="Subscription Key"
                     value={apiKey}
                     onChange={(e) => setApiKey(e.target.value)}
                     disabled={isConnected}
                   />
                 )}
+                { mode === "agent" && (
+                  <Input
+                    placeholder="Entra Token"
+                    value={entraToken}
+                    onChange={(e) => setEntraToken(e.target.value)}
+                    disabled={isConnected}
+                  />
+                )}
+                {/* Entra token input */}
                 {/* Show agent fields if agent mode */}
                 {mode === "agent" ? (
                   <>
                     <div className="space-y-2">
                       <label className="text-sm font-medium">Agent</label>
                     </div>
                     <Input
-                      placeholder="Agent Connection String"
-                      value={agentConnectionString}
-                      onChange={(e) => setAgentConnectionString(e.target.value)}
+                      placeholder="Agent Project Name"
+                      value={agentProjectName}
+                      onChange={(e) => setAgentProjectName(e.target.value)}
                       disabled={isConnected}
                     />
                     {/* Agent ID as Select if agents available, else Input */}
@@ -1299,12 +1313,12 @@ const ChatInterface = () => {
                         disabled={isConnected}
                       />
                     )}
-                    <Input
+                    {/* <Input
                       placeholder="Agent Access Token"
                       value={agentAccessToken}
                       onChange={(e) => setAgentAccessToken(e.target.value)}
                       disabled={isConnected}
-                    />
+                    /> */}
                   </>
                 ) : (
                   <>
@@ -1343,6 +1357,9 @@ const ChatInterface = () => {
                           <SelectItem value="phi4-mm">
                             Phi4-MM Realtime
                           </SelectItem>
+                          <SelectItem value="phi4-mini">
+                            Phi4 Mini (Cascaded)
+                          </SelectItem>
                         </SelectContent>
                       </Select>
                     </div>