Audio completions samples do not work with Phi 4.0 Multimodal Instruct Serverless deployment

[gonsss]

• Package Name:
• Name: azure-ai-inference
• Package Version:
  Version: 1.0.0b9-
• Operating System:
• W11
• Python Version:
• Python 3.10.9 | Anaconda

Describe the bug
I want to run audio completions with Phi-4-multimodal-instruct.
I have followed this example (as it is) and it doesnt work (get 422 Error code)
https://github.com/Azure/azure-sdk-for-python/blob/azure-ai-inference_1.0.0b9/sdk/ai/azure-ai-inference/samples/sample_chat_completions_with_audio_data.py

To Reproduce
Steps to reproduce the behavior:

1. Create a Phi-4-multimodal-instruct serverless deployment on Azure foundry.
2. Run the example https://github.com/Azure/azure-sdk-for-python/blob/azure-ai-inference_1.0.0b9/sdk/ai/azure-ai-inference/samples/sample_chat_completions_with_audio_data.py as it is.
3. I get the following error :
   Error occurred: HttpResponseError
   Error details: (Invalid input) invalid input error
   Code: Invalid input
   Message: invalid input error
   Status code: 422

Expected behavior
Get completion result

Screenshots
N/A

Additional context
N/A

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dargilco @jhakulin @trangevi.

[gonsss]

Indeed, The endpoint i used is Phi-4-multimodal-instruct ( as described in steps to reproduce ) Image processing works great. Audio fails, I have tried multiple wav and mp3 files. Running the example as it is fails. I also tried the c# sdk but it also failed .

…

________________________________ From: Facundo Santiago ***@***.***> Sent: Tuesday, March 4, 2025 3:06:44 AM To: Azure/azure-sdk-for-python ***@***.***> Cc: Gonzalo ***@***.***>; Mention ***@***.***> Subject: Re: [Azure/azure-sdk-for-python] Audio completions sample does not work with Phi 4.0 Serverless deployment (Issue #39928) Hi @gonsss<https://github.com/gonsss>. phi-4 is not a multi-modal model so it can't take audio or images as inputs. The model Phi-4-multimodal-instruct can do it. Error code 422 is telling you that the inputs you are passing to the model are not supported by the model itself. Can you please verify? — Reply to this email directly, view it on GitHub<#39928 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANWCCWSBNVU2FSOA3HIUZIL2SUDDBAVCNFSM6AAAAABYIH66YWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJVHE4TMMBQGA>. You are receiving this because you were mentioned.Message ID: ***@***.***> [santiagxf]santiagxf left a comment (Azure/azure-sdk-for-python#39928)<#39928 (comment)> Hi @gonsss<https://github.com/gonsss>. phi-4 is not a multi-modal model so it can't take audio or images as inputs. The model Phi-4-multimodal-instruct can do it. Error code 422 is telling you that the inputs you are passing to the model are not supported by the model itself. Can you please verify? — Reply to this email directly, view it on GitHub<#39928 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANWCCWSBNVU2FSOA3HIUZIL2SUDDBAVCNFSM6AAAAABYIH66YWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJVHE4TMMBQGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[gonsss]

Just for in case it helps.
I think there is an issue on how the sdks create the http requests.
I have managed to call the REST API (so I skipped using the pythonk sdk).

This is the code I used (removing the keys for obvious reasons)

import requests
import base64
import json
import argparse
import os
from pathlib import Path
import mimetypes

# Configuration
AZURE_ENDPOINT = "YOUR_ENDPOINT"
API_KEY = "YOUR_KEY"
API_VERSION = "2023-12-01-preview"  # Adjust if needed

def encode_audio_file(audio_file_path):
    """
    Read an audio file and encode it as base64.
    Supported formats typically include WAV, MP3, etc.
    """
    with open(audio_file_path, "rb") as audio_file:
        audio_data = audio_file.read()
        return base64.b64encode(audio_data).decode('utf-8')

def get_mime_type(file_path):
    """Get the MIME type of a file based on its extension."""
    mime_type, _ = mimetypes.guess_type(file_path)
    if not mime_type and file_path.lower().endswith('.mp3'):
        return "audio/mpeg"
    elif not mime_type and file_path.lower().endswith('.wav'):
        return "audio/wav"
    elif not mime_type and file_path.lower().endswith('.ogg'):
        return "audio/ogg"
    return mime_type or "application/octet-stream"

def chat_with_audio(system_prompt, user_prompt, audio_file_path, debug=False):
    """
    Send a request to the Azure AI API with system prompt, user prompt, and audio data
    """
    # Prepare headers
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {API_KEY}"
    }
    
    # Prepare the payload
    messages = [
        {"role": "system", "content": system_prompt}
    ]
    
    # Add audio content if provided
    content_parts = []
    if user_prompt:
        content_parts.append({
            "type": "text",
            "text": user_prompt
        })
    
    if audio_file_path:
        try:
            audio_b64 = encode_audio_file(audio_file_path)
            audio_mime_type = get_mime_type(audio_file_path)
            
            # Using the correct content type format according to the error message
            content_parts.append({
                "type": "audio_url",  # Changed from "audio" to "audio_url" as per error message
                "audio_url": {
                    "url": f"data:{audio_mime_type};base64,{audio_b64}"
                }
            })
        except Exception as e:
            return {"error": f"Error processing audio file: {str(e)}"}
    
    # Add user message with content parts
    messages.append({
        "role": "user",
        "content": content_parts
    })
    
    payload = {
        "messages": messages,
        "max_tokens": 1000,
        "temperature": 0.7
    }
    
    if debug:
        print("\nRequest payload:")
        # Print formatted but with truncated audio data to avoid console overflow
        debug_payload = json.loads(json.dumps(payload))
        for msg in debug_payload.get("messages", []):
            if msg.get("role") == "user" and msg.get("content"):
                for part in msg["content"]:
                    if part.get("type") == "audio" and part.get("audio_url", {}).get("url", "").startswith("data:"):
                        # Truncate the base64 data for display
                        url = part["audio_url"]["url"]
                        prefix = url[:30]
                        suffix = url[-10:]
                        part["audio_url"]["url"] = f"{prefix}...{suffix} [TRUNCATED]"
        print(json.dumps(debug_payload, indent=2))
        print("\nRequest headers:")
        print(json.dumps(headers, indent=2))
    
    # Make the API call
    endpoint = f"{AZURE_ENDPOINT}/v1/chat/completions"
    try:
        response = requests.post(endpoint, headers=headers, json=payload)
        if debug:
            print(f"\nResponse status: {response.status_code}")
            print(f"Response headers: {dict(response.headers)}")
        
        if not response.ok:
            error_detail = "Unknown error"
            try:
                error_json = response.json()
                error_detail = json.dumps(error_json, indent=2)
            except:
                error_detail = response.text or "No error details available"
            
            return {
                "error": f"{response.status_code} {response.reason} for url: {response.url}",
                "error_details": error_detail
            }
        
        return response.json()
    except requests.exceptions.RequestException as e:
        return {"error": str(e)}

def main():
    parser = argparse.ArgumentParser(description='Call Azure AI API with system prompt, user prompt and audio')
    parser.add_argument('--system', type=str, default="You are a helpful assistant that can understand both text and audio.", 
                        help='System prompt')
    parser.add_argument('--user', type=str, default="", 
                        help='User text prompt')
    parser.add_argument('--audio', type=str, required=True, 
                        help='Path to audio file')
    parser.add_argument('--debug', action='store_true', 
                        help='Enable debug mode to show request details')
    
    args = parser.parse_args()
    
    if not os.path.exists(args.audio):
        print(f"Error: Audio file {args.audio} not found")
        return
    
    print(f"Sending request with:")
    print(f"System: {args.system}")
    print(f"User: {args.user}")
    print(f"Audio: {args.audio} ({get_mime_type(args.audio)})")
    
    response = chat_with_audio(args.system, args.user, args.audio, args.debug)
    print("\nResponse:")
    print(json.dumps(response, indent=2))
    
    # Extract and print just the assistant's message if available
    if "choices" in response and response["choices"]:
        assistant_message = response["choices"][0].get("message", {}).get("content", "")
        print("\nAssistant's response:")
        print(assistant_message)

if __name__ == "__main__":
    main()

[gonsss]

@trangevi I was wondering if you could make any findings? Anything I can help with?

[trangevi]

Hi @gonsss , the python SDK hasn't been updated yet to support passing in audio via a url, which is currently the only way in which Phi accepts audio inputs. I'm actively working on fixing the Python SDK, and I believe the Phi team is working to accept encoded audio data in the body of the request.

I saw you mentioned that you tried the C# SDK and it also failed, that one however should work. Was it the same error? Would you be able to provide the code you attempted to run there, or was it just the sample?

[gonsss]

Thanks @trangevi ,
Please note the Phi (or rather Azure Foundry serverless endpoint ) already support passing an encoded audio in the body. See my previous comment with an example calling the REST API where I am doing exactly this.

This is what I was testing : use the python / c# sdk to pass an encoded audio in the body, which in both sdks seem to fail with Phi 4.0 multi-modal instruct.

Regarding c# I attached the program I used on this message . Please bear in mind that in both cases I was testing to pass the content encoded in the URL.

In c# case , I have spotted inconsistencies on the sample file : https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/ai/Azure.AI.Inference/samples/Sample12_ChatCompletionsWithAudio.md.

Second sample "tries" to show how to pass the content in the request, but then in the request passes an URL .

In case of Python, I would expect this sample to just work :
https://github.com/Azure/azure-sdk-for-python/blob/azure-ai-inference_1.0.0b9/sdk/ai/azure-ai-inference/samples/sample_chat_completions_with_audio_data.py

Hope this helps.

Program.cs.txt