read_traces() dumps a super long text and floods the context. This API needs redesign.

[KaminariOS]

5830ef09-c739-4bd4-a4cc-65023b65fd84_1758781933.3274791.json

ReAct(with Openrouter/GPT-4-mini)

Problem_id: pod_failure_hotel_res-detection-1 (this is just one example; the same issue happens in other problems too).

As shown in the trace below, the long trace makes the agent completely forget the initial instructions.

Agent action

get_traces("test-hotel-reservation", 10)
read_traces("/proj/aiopslabeval-PG0/AIOpsLab/trace_output/traces_1758815340.csv")
# Dumps a super long text 
        {
            "role": "assistant",
            "content": "It seems you've provided a large dataset related to traces of various operations, likely from a distributed system or microservices architecture. Each entry includes identifiers for trace and spans, service names, operation names, timestamps, durations, error flags, and response codes. \n\nIf you need help analyzing this data or extracting specific insights, please specify what you would like to know or what kind of analysis you're interested in. Here are a few examples of what I can help you with:\n\n1. **Error Analysis**: Identify any operations that resulted in errors.\n2. **Performance Metrics**: Calculate average durations for specific operations or services.\n3. **Service Dependency Mapping**: Analyze the relationships between services based on the trace data.\n4. **Response Codes**: Summarize the response codes for different operations.\n\nLet me know how you'd like to proceed!"
        },
        {
            "role": "env",
            "content": "Error parsing response: \nFormat validation failure. Only have one pair of three ticks in your block and check the ticks. \nCorrect example 1:\nI should run:\n```\nls\n```\n\nCorrect example 2:\nCheck k8s info \n```\nkubectl get services --all-namespaces\n```\n            "
        },
        {
            "role": "assistant",
            "content": "Thought: The previous output indicates that there was an issue with the formatting of the code block, specifically that it needs to have three backticks at the beginning and end to properly format the content as a code block.\n\nAction: I will ensure to format any future code snippets correctly with three backticks to avoid parsing issues. Here\u2019s the corrected format:\n\n```\nYour code or command here\n```"
        },
        {
            "role": "env",
            "content": "Error parsing response: No API call found!"
        },
        {
            "role": "assistant",
            "content": "Thought: The error indicates that there was an expectation for an API call in the response, but none was found. This suggests that the response may not be in the correct format or that the necessary information was missing.\n\nAction: I will review the previous context or data to ensure that an appropriate API call is included in the next response. If needed, I will provide an example of a valid API call to meet the expected format."
        },
        {
            "role": "env",
            "content": "Error parsing response: \nFormat validation failure. Only have one pair of three ticks in your block and check the ticks. \nCorrect example 1:\nI should run:\n```\nls\n```\n\nCorrect example 2:\nCheck k8s info \n```\nkubectl get services --all-namespaces\n```\n            "
        },

[HacksonClark]

Yes, we're well aware of how long the traces context is. The point of the tool is just to provide the information. For implementation in an agent, it may make more sense to summarize first to not flood the context.

[KaminariOS]

Yes, we're well aware of how long the traces context is. The point of the tool is just to provide the information. For implementation in an agent, it may make more sense to summarize first to not flood the context.

The summary can be expensive. A few long traces like this cost me ~ $1(about 120k tokens) last night when I ran 56 problems.

[HacksonClark]

I understand, though I don't view this as much of an issue on the benchmark side.

[KaminariOS]

Instead of read_traces, search_in_traces aligns closer to how human engineers work.

More generally, the agent should be aware of the budget and file size, and generate Python script or use commands like tail, grep, head on the fly to process the data when it is too big.

[HacksonClark]

the agent

Yes, exactly. My point is that the agent should do this, the benchmark doesn't need to. You can wrap the tool (or implement your own) however you wish. I think this approach makes a lot of sense, and I would merge a good implementation of this as a tool. However, I don't think this is something that needs to be implemented by the benchmark.

[KaminariOS]

Sure, it is part of the Agent Cloud Interface, not the benchmark.

This is an infrastructure that all agent needs.

[HacksonClark]

@KaminariOS I think your search_in_traces idea could be implemented alongside what we currently have, which is my point earlier.

[KaminariOS]

I notice that kubectl get events also dumps a super long text(300K), although not as long as traces(1MB).