[Bug]: chat_result.cost for Swarm Agents not working

[bassilkhilo]

Describe the bug

When I try to get the cost of the chat, it returns the default:

usage_including_cached_inference = {"total_cost": 0}
usage_excluding_cached_inference = {"total_cost": 0}

Steps to reproduce

I think I've identified the issue.

in utils.py the problematic code is:

for agent in agents:
    if getattr(agent, "client", None):
        aggregate_summary(usage_including_cached_inference, agent.client.total_usage_summary)
        aggregate_summary(usage_excluding_cached_inference, agent.client.actual_usage_summary)


I printed the agent in the above for loop, and I see:

<autogen.agentchat.user_proxy_agent.UserProxyAgent object at 0x0000026812CED070>
<autogen.agentchat.groupchat.GroupChatManager object at 0x0000026812CED0D0>

There is no client attr and thus it will always skip the if statement that aggregates the summary. Why, not sure still investigating.

Model Used

gpt-4o-2024-08-06

Expected Behavior

No response

Screenshots and logs

No response

Additional Information

No response

[marklysze]

@Hk669, are you able to help on this one :)

[Hk669]

@bassilkhilo can you please share the complete code, to reproduce this issue. and as suggested, i think it makes sense that only client is taken into consideration, but let me reproduce before jumping into the PR. Thanks

[bassilkhilo]

Hello @Hk669

I mean it doesn't work for the example code provided in one of the notebooks:

`import json
import os
from autogen import (
SwarmAgent,
SwarmResult,
initiate_swarm_chat,
AFTER_WORK,
AfterWorkOption,
ON_CONDITION
)

from dotenv import load_dotenv

load_dotenv()

main_openai_model_for_agents = "gpt-4o-2024-08-06"

secondary_openai_model_for_generate_reply_cases = "gpt-4o-mini-2024-07-18"

openai_api_key = os.environ.get("OPENAI_API_KEY")
bing_api_key = os.environ.get("BING_API_KEY")

from agents.utils.autogen_initialization_utils import get_llm_config

llm_config = get_llm_config(main_openai_model_for_agents, openai_api_key)

Define functions used by the swarm agents

def verify_customer_identity(passport_number: str, context_variables: dict) -> SwarmResult:
context_variables["passport_number"] = passport_number
context_variables["customer_verified"] = True
return SwarmResult(value="Customer identity verified", context_variables=context_variables)

def approve_refund_and_transfer(context_variables: dict) -> SwarmResult:
context_variables["refund_approved"] = True
return SwarmResult(value="Refund approved", context_variables=context_variables, agent="PaymentProcessor")

def process_refund_payment(context_variables: dict) -> SwarmResult:
context_variables["payment_processed"] = True
return SwarmResult(value="Payment processed successfully", context_variables=context_variables)

Configure Swarm Agents

customer_service = SwarmAgent(
name="CustomerServiceRep",
system_message="You are a customer service representative. Verify identity and transfer to refund specialist.",
functions=[verify_customer_identity],
llm_config=llm_config # Replace with your desired model config
)

refund_specialist = SwarmAgent(
name="RefundSpecialist",
system_message="You are a refund specialist. Approve refund and transfer to payment processor.",
functions=[approve_refund_and_transfer],
llm_config=llm_config
)

payment_processor = SwarmAgent(
name="PaymentProcessor",
system_message="You are a payment processor. Process refund payment and notify the customer.",
functions=[process_refund_payment],
llm_config=llm_config
)

Define handoffs

customer_service.register_hand_off(
hand_to=[
ON_CONDITION(refund_specialist, "After customer verification, transfer to refund specialist"),
AFTER_WORK(AfterWorkOption.REVERT_TO_USER)
]
)

refund_specialist.register_hand_off(
hand_to=[
AFTER_WORK(payment_processor)
]
)

payment_processor.register_hand_off(
hand_to=[
AFTER_WORK(AfterWorkOption.TERMINATE)
]
)

Initialize Swarm Chat

context_variables = {
"passport_number": "",
"customer_verified": False,
"refund_approved": False,
"payment_processed": False
}

chat_result, final_context, last_agent = initiate_swarm_chat(
initial_agent=customer_service,
agents=[customer_service, refund_specialist, payment_processor],
messages="Customer requesting refund for order #12345",
context_variables=context_variables,
after_work=AFTER_WORK(AfterWorkOption.TERMINATE)
)

Output the results

print("Chat Result:")
print(chat_result)

print("\nFinal Context Variables:")
print(json.dumps(final_context, indent=2))

print("\nLast Agent:")
print(last_agent)
`

[Hk669]

I'm not sure about the swarm agents, maybe it's the solution that you've proposed, let me try if it's working. Thanks! @bassilkhilo

[bassilkhilo]

Hi @Hk669 hope all is well! Was wondering if you had any new insight as to why it's not working, thank you!

[marklysze]

Hi @bassilkhilo, just a note that I can replicate this issue. Investigating...

[Hk669]

thanks @marklysze! i couldnt get the time to replicate this issue locally. let me try pulling the PR locally and test it with all the types of chats once it is ready for review.

[marklysze]

thanks @marklysze! i couldnt get the time to replicate this issue locally. let me try pulling the PR locally and test it with all the types of chats once it is ready for review.

No problem @Hk669, I'm working with PR #173 for the Group Chat underlying issue. I'll have a look at nested chat as well, though that may be a separate PR.

Note that one on one chats using initiate_chat were okay.

[bassilkhilo]

Thank you team for looking into this!

[marklysze]

@bassilkhilo and for anyone looking into this. We think incorporating telemetry is now a better option because:

• it reduces the complexity of managing costs for multiple providers
• allows new orchestrations, and multi-level chats (e.g. nested chats), to operate without having to manage the token usage/cost bubbling up through them.
• support temporary agents, such as the group chat speaker selection agents, that don't have a life-cycle that allows us to get their token and cost usage

So, to that end, we have started the process working out what Telemetry looks like. Broadly speaking we will support OpenTelemetry as well as build a protocol so other telemetry providers can be created.

For this Issue and the related PR, 173, we will leave them open for now until telemetry is in place and if anyone wants to incorporate the changes into their code base they can.

[aa-wong]

What is the status of this issue? Is this still an open good first issue to work on or is it now no longe needed because telemetry solution is in process