Question about collecting eval requirements [Paper]

[CrisRodriguez]

Hello!

I’ve been reading the Agent-as-a-Judge paper and found it quite interesting. Thanks for the work!

One thing I’m unclear about is how this framework would generalize to real-world queries that don’t come with predefined requirements or dependencies (unlike in the DevAI benchmark).

The paper doesn’t go into much detail on this. How do you think requirements could be collected or inferred in a more open-ended, practical setting? Would it be an additional agent step, or maybe structured prompting?
Curious to hear what you think about this and if you have run some experiments.

Thanks,

[mczhuge]

Hi, I really appreciate your interests. I want to share some review during our submission.

How generalizable is the Agent-as-a-Judge framework beyond DevAI's code generation tasks, and how could it be adapted to other domains.

We focused on code generation as it has clear intermediate objectives and is a promising application of agentic systems, but the concept of Agent-as-a-Judge is domain-agnostic. Even the modular implementation of our version of Agent-as-a-Judge (e.g. graph parsing, search, read, etc.) already support multiple data modalities, including code, text, images, and videos. We expect that extending Agent-as-a-Judge to other domains might just require one to potentially recombind our modules (potentially adding more functions their own).

As you'll be aware, the field of agentic systems is moving quite quickly. Indeed, several other papers have already directly implemented agent-as-a-judge for other applications, like mobile apps [1], long documents [2], bioinformatics [3], github issue [4]:

[1] AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents

[2] Agent-as-Judge for Factual Summarization of Long Narratives

[3] BioAgents: Democratizing Bioinformatics Analysis with Multi-Agent Systems

[4] Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution