How does OrchestrationHandoffs manage threads and chat history?

[rmbayer]

Python==3.10
Semantic-kernel==1.37.0
Language Model = gpt-4o

Details of Agents at the end in Project Details

Description:
I'm having issues getting Orchestrations to perform as I would expect. Specifically, I am now very involved with the OrchestrationHandoffs() utilizing a ChatCompletionAgent().

I have agents that I have built from ChatCompletionAgent() using an AzureChatCompletion() service. Alone these agents work well. They each have their own Kernel and ChatHistoryAgentThread.
Now I am trying to have an orchestrator utilize handoffs and am getting hallucinations and less desirable results.

Example 1:
If I say "show me 20 records for New York", the SQL agent alone will use it's system instructions to lookup the code for New York and create and run a SQL query, then explain a summary of the data to the user.
I can then keep the conversation going and say, I also only want to look at Brooklyn, and the agent will keep the same "SELECT TOP 20... Where State=" and also add the appropriate "AND CITY="

When I now wrap this in a OrchestrationHandoffs() agent I see multiple issues:

• Orchestration agents use a InProcessRuntime and do not have a chat history, so follow-up questions are treated as completely new questions.
• The Orchestration agent does not use the response from the SQL Agent, instead all the user sees is "Task is completed with summary: No handoff agent name provided and no human response function set. Ending task."
• Now the underlying agents are making mistakes and hallucinating column names that don't exist and are trying to pass arguments that don't exist to functions

Example 2:
If I say "Show me 20 records for new york and create a graph", it will call the SQL agent to pull the data and produce a graph, but then it just says "here is the graph".
Calling the SQLAgent on it's own would say "I pulled the data, I have records for ... and they look like ..." and then if those were saved to a .csv and I called the GraphAgent on it's own it would say "I created a graph visualizing xyz"

Questions I have:

• How is a conversation thread managed with a OrchestrationHandoffs() agent? Can I access a ChatHistory() of this agent?
• Does the orchestration directly just push the prompt to the other agents or does it reformat it?
  • The only thing I can see in logs is INFO:semantic_kernel.kernel:Calling Handoff-transfer_to_SQLAgent function with args: {}
• Is the orchestration directly calling the underlying tools? or is it calling the agent.invoke()?
• How can I manipulate the Chat conversation to show the final results of each agent before a handoff?

Project Details:

4 Agent system

• Orchestrator
• Sql agent
• Graph agent
• Analyzer agent

SQL Agent: Has database Schema, join relationships, examples, and a set of tools. It's role is to pull the necessary data for other operations, or just answer basic questions.
Tools:

• generate_and_execute_sql
• get_available_location
• load_more_records
• get_sample_data
  Alone, this agent works great when just using .invoke(). It gets queries correct, explains the data it retrieved, and stores a pd.dataframe in memory.

Graph Agent: Retrieves the records pulled by the SQLAgent and can create graphs based on examples it has in it's prompt
Tools:

• create_matplotlib_graph
  Again, this agent alone works great by loading a .csv or a pandas dataframe

Analyzer Agent: Has specific pd.dataframe operations that it performs based on the data available to understand what data there is.

• analyze_dataframe
  Again, this agent alone works great by loading a .csv or a pandas dataframe