add deep research agent benchmark script.

[lkk12014402]

Description

add benchmark script for evaluating the accuracy of deep research agent.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

• DeepResearchAgent/requirements.txt

[Copilot]

Pull Request Overview

Adds a benchmarking harness for measuring the accuracy of the Deep Research Agent by integrating dataset loaders, an LLM-based scoring function, and CLI support.

• Wraps the agent’s raw output in a JSON field ("answer") in research_agent.py.
• Introduces eval.py to load questions, invoke the agent, score answers via an LLM judge, and save results.
• Updates documentation under benchmark/accuracy and the main README to cover setup and usage.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
DeepResearchAgent/research_agent.py	Return payload now wrapped as {"answer": ...}
DeepResearchAgent/benchmark/accuracy/eval.py	New benchmark script for accuracy evaluation
DeepResearchAgent/benchmark/accuracy/README.md	Instructions for running the accuracy benchmark
DeepResearchAgent/README.md	Added note on configuring deep_researcher.yaml in setup

Comments suppressed due to low confidence (2)

DeepResearchAgent/benchmark/accuracy/eval.py:93

• [nitpick] There are no unit tests covering key functions like load_questions, process_single_question, or run_benchmark. Adding tests will help ensure correctness when loading datasets and computing accuracy.

def load_questions(dataset_names: list[str] | None = None) -> list[dict[str, str]]:

DeepResearchAgent/benchmark/accuracy/eval.py:334

• The args.agent_config field is referenced in metadata but no --agent-config argument is defined in the parser. Consider adding a corresponding parser argument or renaming this field to match --service-url or another existing flag.

            "agent_config": args.agent_config,

[minmin-intel]

Can you add some benchmark results in README? And add some recommendations about what LLMs to use with vllm-gaudi?

[lkk12014402]

Can you add some benchmark results in README? And add some recommendations about what LLMs to use with vllm-gaudi?

no problem. I will

[lkk12014402]

Can you add some benchmark results in README? And add some recommendations about what LLMs to use with vllm-gaudi?

add togethercomputer/together-search-bench benchmark accuracy data

[joshuayao]

Hi @lkk12014402 could you please help check the CI failures? Thanks.