Foundry hosted agents + Service managed chat history + Aspire , how to make it work?

[Matthew Paul (mip1983)]

I'm experimenting with the combination in the title:

Aspire app host, which wires up Azure foundry and a .NET hosted agent. The initial goal is just to have a single agent with service managed conversation history, with an aim to have something where the client only sends the users chat message (and perhaps contextual information), and the agent uses the service managed conversation history. I then want to be able view previous threads and recall a chat session, much like you can with co-pilot in VS or in ChatGPT. So nothing to crazy for now, just a kind of agent hello world but with a foundry hosted agent and managed chat history that I can recall.

So I'm picturing on the client side calling the agent, it should just be able to pass an identifier for the conversation and any new message(s), and the agent take care of the conversation history via foundry/service.

However, I'm getting very confused in figuring out how to get this combination working. I've put my test solution here: https://github.com/ecoDriverltd/FoundryAgentsExperiment

I'm trying out various combinations of things in the test project: Test repo

I've read this and see these different modes of operation: Agent framework chat history storage patterns
So in my testing I'm trying to figure out how you get mode 1 or mode 3 to get service managed storage.

However, whatever I try, I don't seem to end up with something that works with the foundry managed chat history. I either get an in-memory chat history or I can't figure out how to make the client library work or what it should be talking to.

Some of my points of confusion are:

• I register my agent with aspire and use 'AsHostedAgent' passing my foundry project resource. The code summary for this methods says:
  'Configures the resource to run and publish as a hosted agent in Microsoft Foundry, targeting the specified Foundry project', however it seems to run locally. i.e. if I do 'app.GetEndpoint("agent-dotnet")', I get a localhost url, not a url to an agent in foundry. Will foundry managed chat history work if the agent is running locally?

• I've tried 'AGUIChatClient' in my 'AGUIAgentRecallsFact' test. I've tried using the foundry project client to create a conversion and then use the conversation id in the 'AgentSession', but that conversation id seems to be doing nothing. It doesn't recall history/context if I start a new session with the same conversation id, I think it's just using a new in memory chat history provider for the session.

• I've tried to create a 'FoundryAgent' (FoundryAgentRecallsFact_ConversationSession test), but trying to do this using the project client, but pointing at the agent running locally, it's just not happy (Agent 'agent-dotnet' not found). Foundry wants to be hosting the agent for this client and presumably the managed chat history to work, but to my first point, aspire runs it locally. I thought maybe the whole project client wants to point at the local agent so conversations are established there, and I've tried various things on the agent side like 'MapOpenAIConversations' to see if that would work, but no joy, the project client is only happy talking to the azure foundry project.

I've seen this blog: Multi agent systems with aspire and agent framework

Which has various bits in the code to make the local agent lets say 'Foundry compatible', which I've borrowed to no effect. In the source for that, it's doing some custom stuff with react to talk to the agent and pass through some conversation identifier I think. There seems to be quite a lot of hacks/work around and custom code to make that AlpineAI ski-resort demo work, not something I'd want to bring into production code. I'd have thought I should be able to use a .NET agent framework client rather than try and emulate what's being done in react there.

tbh It feels like this stuff is perpetually in preview, should I forget about foundry managed chat history for now and look at plugging in my own ChatHistoryProvider? (In my production app today, I have semantic kernel and use that to store/manage conversation history in cosmosDb, so now exploring options with agent framework and foundry).

[David Fowler (davidfowl)]

cc Tommaso Stocchi (@tommasodotNET)

[Tommaso Stocchi (tommasodotNET)]

Hi Matthew Paul (@mip1983), so the hosted agent runs locally and this is expected. You only get a foundry agent endpoint when you deploy, since foundry hosted agents are basically just a compute resource.

They way we intended the implementation, it made sense to have the agent run locally until deployment, as it were a simple api project.
They history management handle by foundry will actually work after deployment.

As per the frontend, I'd recommend going to foundry, in the deployed agent section, where you have a full working sample of a frontend that interacts with your deployed agent (with link to source code).

The questions about chat history storage patterns I think might be better answered in the microsoft/agent-framework repo.
We've tried to keep our implementation as abstract as possible, my sample usually leverage agent framework, but you can build hosted agents in different ways and Aspire should behave in the same way despite the way you write your agent.