When can I have a conversation about uploaded documents with the model within a chat?

[ystvan]

Hi Team,

I can see a lot of exciting deliverables in #7126

My general question: is it possible as of now to send a file as a BinaryContent?

Currently my rudimentary code throws System.NotSupportedException: 'Unsupported chat message content type 'Microsoft.SemanticKernel.BinaryContent'.'

Does not matter which version

<PackageReference Include="Microsoft.SemanticKernel" Version="1.18.1-rc" />
Or
<PackageReference Include="Microsoft.SemanticKernel" Version="1.17.2" />

var kernelBuilder = Kernel.CreateBuilder();

kernelBuilder.AddAzureOpenAIChatCompletion(
         "gpt-4o",
         "https://myservice.openai.azure.com/",
         "myKey");

// Create kernel
var kernel = kernelBuilder.Build();


// Get chat completion service
var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

ChatHistory history = [];
history.AddSystemMessage("You're a friendly and helpful assistant that responds to questions directly");
                            
string pdfPath = "https://my.blob.core.windows.net/files/myDocument.pdf";
var uri = new Uri(pdfPath);

var message = new ChatMessageContentItemCollection
{
    new TextContent("Can you tell me what's in this document?"),
    new BinaryContent(uri)
};

 history.AddUserMessage(message);


OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
{
    MaxTokens = 4000,
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

// Get the response from the AI
var response = chatCompletionService.GetStreamingChatMessageContentsAsync(
               history,
               executionSettings: openAIPromptExecutionSettings,
               kernel: kernel);

await foreach (var msg in response)
{
    Console.Write(msg);
    combinedResponse += msg;
}

// Add the message from the agent to the chat history
history.AddAssistantMessage(combinedResponse);

[evchaki]

@ystvan - check out this sample - https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/OpenAIAssistant_FileManipulation.cs it looks like you are almost on the right track, but you need to use agents vs just passing over a file to a chat completion.

[JP-droidic]

@evchaki To piggyback on this conversation, what is happening with the dotnet-1.18.X? When is it going to be released? I belive you were looking for August 27 as the date

Thanks

[evchaki]

@JP-droidic - last week during office hours we announced it would be this week. The release should be out tomorrow. There were additional items we wanted to test to make sure no additional issues were found.

[ystvan]

@ystvan - check out this sample - https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Agents/OpenAIAssistant_FileManipulation.cs it looks like you are almost on the right track, but you need to use agents vs just passing over a file to a chat completion.

Thank you for the pointers @evchaki . I can see this is using a service OpenAIFileService File service access for OpenAI: https://api.openai.com/v1/files do we have feature parity in Azure OpenAI?

I do not have access to OpenAI models directly, I use the services only via Azure OpenAI

[JP-droidic]

@JP-droidic - last week during office hours we announced it would be this week. The release should be out tomorrow. There were additional items we wanted to test to make sure no additional issues were found.

Amazing, thank you for letting me know, @evchaki - I was looking for a recording of last week's office hours, but they are still not uploaded to YouTube. Anyway cheers

[crickman]

Hi @ystvan - Today we are merging support for V2 assistants into main and releasing a new package. At this point in time, I don't believe chat-completion supports the specification of uploaded files by id (https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages). Images may be specified by an image "url".

I believe uploading files is specific to the assistant API...which @JP-droidic correctly noted that we've been working to support V2 features. This V2 support has been available in the 1.18.0RC and 1.18.1RC packages and is being merge to main today.

An updated version of the "file-manipulation" sample Evan referenced can be found here (until the merge is complete, then it will be in main): https://github.com/microsoft/semantic-kernel/blob/feature-connectors-openai/dotnet/samples/Concepts/Agents/OpenAIAssistant_FileManipulation.cs

[ystvan]

Hi @crickman

Great news with today's merge! I did some successful experiments with Evan's reference (using Azure OpenAI models and endpoints) and noted some warnings, like

but could not find any replacement in Azure.AI.OpenAI library for the OpenAIFileService so far.

I don't know who could answer my lame questions:

I would like my custom Copilot/Chatbot to have the capability of having a conversation about uploaded files like images, PDFs or Word docs.

• Can agents/assistants access Plugins? I can see the agent's line-of-tought, such as using Optical Character Recognition (OCR), attempting to extract text, etc. Would it be better to have the Kernel access to a OCR Plugin and my assistant/agent could use it if it needs it?
  OR
• Should I have multiple agents, one maybe executing OCR, another some Azure Document Intelligence tasks, another on text/translation and have them orchestrated by the Kernel to work towards a common goal?

I checked out the branch feature-connectors-openai and building it at the moment to see the updated version of the test.

[ystvan]

Closing this and open a new thread to stay on topic, thank you @evchaki @crickman both for finding a solution! I really do appreciate the magic of Q&A.