How to use tools with parameters in a group chat

[jmerkow]

I have a few agents that I want to use tools and to support a discussion, but i am unsure how to pass parameters to the tool functions in a group chat. I am using 0.4.6.

Say I have a these tools and agents.

def clinical_findings_tool(data: dict) -> list:
    """
    Simulate generating clinical findings based on image analysis and free-text extraction.
    Expects input in the form: {"data": {"analysis": ..., "extracted": ...}}
    """
    print("clinical_findings_tool", data, "~1~!~!~!~!~!")
    # FAKE will need to use data to get this!
    findings = [
        'The heart size is normal.',
        'The aorta is tortuous.',
        'A patchy infiltrate in the right lower lobe is associated with volume loss.',
        'A small right pleural effusion is present.',
        'The left hemithorax is grossly clear.'
    ]
    return findings
# Clinical Findings Agent
clinical_findings_agent = AssistantAgent(
    name="ClinicalFindingsAgent",
    system_message="You are an agent that generates clinical findings based on image analysis and exam text. When discussing, please use your tool to generate findings.",
    model_client=model_client,
    tools=[clinical_findings_tool],
    reflect_on_tool_use=True
)

def image_search_tool(data: dict) -> list:
    """
    Simulate searching for similar radiology reports based on image data.
    """
    print("image_search_tool", data, "~1~!~!~!~!~!")
    # FAKE will need to use data to get this!
    reports = [
        "Report 1: Consistent with findings of mild pneumonia.",
        "Report 2: The x-ray suggests a small effusion.",
        "Report 3: Normal chest x-ray with no significant abnormalities."
    ]
    return reports
    
# Image Search Agent
image_search_agent = AssistantAgent(
    name="ImageSearchAgent",
    system_message="You are an agent that searches for similar radiology reports based on image data and clinical findings. When discussing, please use your tool to report similar findings and note any uncertainties.",
    model_client=model_client,
    tools=[image_search_tool],
    reflect_on_tool_use=True
)

discussion_prompt = (
        "You are a team of experts. "
        "The Clinical Findings Agent should call its clinical findings tool to generate detailed findings from the image data. "
        "The Image Search Agent should call its search tool to retrieve similar radiology reports. "
        "Review these details and then discuss which findings are most reliable and which are uncertain. "
        "Finally, produce a final consensus summary that begins with 'CONSENSUS:' followed by your agreed summary."
    )
    
termination_condition = TextMentionTermination("CONSENSUS:") | MaxMessageTermination(max_turns)

team = MagenticOneGroupChat(
    participants=[clinical_findings_agent, image_search_agent],
    model_client=model_client,
    termination_condition=termination_condition,
    max_turns=max_turns,
    max_stalls=max_stalls,
    final_answer_prompt="Generate a consensus summary that begins with 'CONSENSUS:'"
)

# DOESN"T WORK
#prompt = MultiModalMessage(content=[discussion_prompt, extra_data], source="user")
prompt = discussion_prompt

results = await Console(team.run_stream(task=prompt))
    

Essentially, I want these agents to be able to make the above function calls, but it's unclear go to get them to format the data for the input.

FYI. I am also on Teams, feel free to reach out there, or grab time on my calendar. [email protected]

[ekzhu]

As we have discussed, the key is to make the argument and return types simpler:

1. use flattened list of arguments rather than a dictionary
2. use string return type rather than a list, with well formed, natural language string formatted with the output of the code.
3. Set description for each agent to make it easier for the group chat to choose