How to retrieve file_id when streaming a response from an agent?

[GabrelBulz]

I'm curious if there's a way to retrieve a file_id (not of type .png or any other image type) when streaming a response from an agent.

For example, if I ask the agent (using the code interpreter) to create a plot pie with some random data, the resulting file_id (containing the generated image) is available in the response object under response.content.items as an object of type StreamingFileReferenceContent.

However, if I ask the model to generate another file type (such as a text file containing 5 random names) the file_id does not appear in any of the chunks yielded by the streamed response.

Is there a way to access the file_id in such cases?

[moonbox3]

Hi @GabrelBulz, the file_id handle should be included as part of the StreamingAnnotationContent's file_id attribute. When you can StreamingChatMessageContent back from the agent, check for that StreamingAnnotationContent item type inside of the items collection.

By the way the reference file_id I am assuming you're working with SK Python (please correct me if I am wrong). You may need to use the on_intermediate_message callback to get a handle to intermediate messages, if the annotation content isn't returned with the final response.

Which agent type are you working with?

[GabrelBulz]

I'm using OpenAI Agents with python sdk.

When using on_intermediate_message, the file_id is indeed present inside the StreamingAnnotationContent.
However, this item (which contains the file_id) does not appear in the AgentResponseItem.

Is this the intended behavior?

[moonbox3]

To clarify, you're using the Azure/OpenAI Assistant Agent*, correct? We don't have any support for OpenAI Agents specifically (from their Agents SDK).

The reason why we introduced the on_intermediate_message callback is to provide a way to get the intermediate messages produced during invocation, that are not the final message. The AgentResponseItem is considered a "final message" construct -- so for the streaming case there are many chunk messages produced and returned to the caller.

We've got a fix in main that properly handles the order of the intermediate messages -- right now, they FunctionCallContent/FunctionResultContent types are returned after the final message (when using function calling). Just a note.

[GabrelBulz]

Thanks for the clarification