Question on evaluation integrity: Is full-document ingestion into experience base permissible?

[echo-yiyiyi]

Thank you very much for your work. this benchmark is highly intriguing.

After reading your paper, my understanding is that you intend to withhold the original documentation from the models/agents during the testing phase. Instead, agents are expected to generate and store their own 'experiences' during training for subsequent retrieval.

However, how do you prevent 'benchmark hacking' regarding the granularity of the stored experiences? For instance, would it be permissible if an agent simply ingests the entire documentation into its experience base? I am curious about the constraints you've implemented to ensure the evaluation remains fair.

[chenweize1998]

Hi, thanks for the great question. You are right, ideally a model with perfect long-context understanding could "hack" the benchmark by ingesting the full documentation as "experience".

Nevertheless, this remains a non-trivial challenge empirically. We tested Qwen3-Coder-480B-A35B-Instruct with the full API doc (~200k tokens) in context, followed by the question and requirements (should only use zwc, etc.), and it only achieved 66% on single-function and 42% on multi-function tasks. The model struggled to adhere to the new zwc library, frequently hallucinating non-existent functions or reverting to numpy.

Still, we acknowledge that as models improve, this will become a critical issue for self-evolution benchmarks. And to be honest, it is a challenge that we not yet perfectly addressed. Would really love to hear if any potential approach can be taken, and would be happy if we can continuously make the benchmark better together.

[echo-yiyiyi]

Thank you for your response.

I believe this benchmark is a fair way to evaluate models or training method through training without documentation.

However, when the benchmark is used to evaluate agents and allows them to summarize their experiences, perhaps we should also allow them to use or copy documentation to ensure fairness?

Additionally, I think it would be very interesting if the paper could present results for 'ACE/EXPEL + Closed-SFT-RL'.

Just a suggestion of mine and might not be entirely correct.

[chenweize1998]

You are absolutely right. I think agents are allowed to use or copy the documentation. Since our goal is a diagnostic test where an ideal agent should achieve near 100% accuracy, successfully identifying, retrieving, and utilizing the correct documentation segment is a completely valid and expected form of knowledge internalization.

Also, combining ACE/EXPEL with our Closed-SFT/RL setup is a good suggestion. We will look into it. Thanks!