Caching in AutoGen V0.4.7

[thrishathi02]

I have a working flow, and I'm using Jupyter notebook to run this flow.

This is the only block of cell that I'll ideally have to re-run by changing the task:
` critic_tet_mention_termination = TextMentionTermination("ALL-GOOD")
terminator_text_mention_termination = TextMentionTermination("max-3-tries")
max_messages_termination = MaxMessageTermination(max_messages=25)

termination = critic_text_mention_termination | max_messages_termination | terminator_text_mention_termination 

team = SelectorGroupChat(
    [planner, retrieval, answer_generator, critic],
    termination_condition=termination,
    model_client = az_model_client,
    selector_prompt=custom_selector_prompt,
    allow_repeated_speaker=False
)

# This will stream the entire conversation to the console.
await Console(team.run_stream(task="Compare 4k TVs of LG and Sony and Micromax"))`

When I run the cell for the first time, everything works according to plan. But when I change the Task and re-run the cell, it throws an error saying 'critic message not found'. This issue is solved when I restart the kernel and run the whole notebook.

Previously I did not have this problem because caching was automaticall implemented in V0.2, is there any way to do it in V0.4 aswell?

[ekzhu]

it throws an error saying 'critic message not found'.

Can you post the exact error message and stack trace?

Could you help me understand why this is related to caching?

For caching: https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.cache.html

[thrishathi02]

Previously, in AutoGen v0.2:
When I ran my workflow, a .cache directory was automatically created.
If I encountered issues related to stale conversation states or missing messages, I could delete the .cache folder before re-running my task.

With AutoGen v0.4, The .cache directory is no longer created (or at least, AutoGen doesn’t seem to rely on it the same way). Now I'm assuming, AutoGen likely caches in memory instead of disk, meaning that old agent states persist between cell executions in Jupyter.

I could solve this issue by using a .py file.

[thrishathi02]

Also, could you also help me understand why I'm getting this warning
"UserWarning: Resolved model mismatch: gpt-4o-2024-08-06 != gpt-4o-2024-11-20. Model mapping in autogen_ext.models.openai may be incorrect."

[ekzhu]

The warning is because the model mapping, we have in the code doesn't match what your API returns. Are you using Azure Open AI and using your deployment name for the model field in AzureOpenAIChatCompletionClient? You should be using azure_deployment field for that instead and use the actual model name for the model field.

[ekzhu]

We don't automatically cache model output in v0.4. To cache, use https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.cache.html

[thrishathi02]

I'm assuming that if I use the cache_client in an agent, The token consumption will reduce?

I'm trying to do this, and the token consumption somehow seems to increase.

`
async def main():

with tempfile.TemporaryDirectory() as tmpdirname:
    cache = Cache(tmpdirname)
    cache_store = DiskCacheStore[CHAT_CACHE_VALUE_TYPE](cache)
    cache_client = ChatCompletionCache(az_model_client, cache_store)


   # Metric Conversion Agent
    metric_agent = AssistantAgent(
        name="metric_agent",
        model_client=cache_client,
        system_message="Convert the user's input into metric units. Respond only with the numeric result. Respond with 'TERMINATE- 
        AGENT' once you are done with your task.",
    )

    task = "I am 5 feet tall"
    # First call - response from OpenAI
    result1 = await Console(metric_agent.run_stream(task=task))
    print(result1)

    result2 = await Console(metric_agent.run_stream(task=task))
    print(result2)

    cache.close()

`

Or does the LLM caching only work when I use something like this?
response = await cache_client.create([UserMessage(content="Hello, how are you?", source="user")])

[ekzhu]

# First call - response from OpenAI
result1 = await Console(metric_agent.run_stream(task=task))
print(result1)

result2 = await Console(metric_agent.run_stream(task=task))

These are not independent calls, as agent is stateful and you are passing the message history of the first call in the second call.

Read about agent in tutorial: https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/tutorial/agents.html#getting-responses

[thrishathi02]

So, if I call the on_reset() method on the agent, and then use the cache_client, then a separate LLM will not be made for the same task, rather the cached response would be used?

[ekzhu]

For AssistantAgent this should be the case. Depending on how the agent implements the on_reset method.

[thrishathi02]

How would I know if it is a cached response and not a seperate LLM call?

Is there a separate variable or property that is identifiable?