Skip to content

[Bug]: evaluating_adk_agent uses predicted_trajectory instead of reference_trajectory #2370

@stepanl-bsy

Description

@stepanl-bsy

File Name

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluating_adk_agent.ipynb

What happened?

In the Prepare Agent Evaluation dataset of the evaluating_adk_agent notebook, there is a critical typo:
predicted_trajectory should be reference_trajectory

eval_data = {
    "prompt": [
        "Get price for smartphone",
        "Get product details and price for headphones",
        "Get details for usb charger",
        "Get product details and price for shoes",
        "Get product details for speaker?",
    ],
    "predicted_trajectory": [. # should be reference_trajectory
        [
            {
                "tool_name": "get_product_price",
                "tool_input": {"product_name": "smartphone"},
            }
...

compare it with the correct eval_data in this notebook evaluating_langgraph_agent:


eval_data = {
    "prompt": [
        "Get price for smartphone",
        "Get product details and price for headphones",
        "Get details for usb charger",
        "Get product details and price for shoes",
        "Get product details for speaker?",
    ],
    "reference_trajectory": [
        [
            {
                "tool_name": "get_product_price",
                "tool_input": {"product_name": "smartphone"},
            }
        ],

Relevant log output

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions