No multi-turn user interactions observed in Gaia2 HF traces

[cyche23]

Description

According to the Gaia2 paper, the benchmark appears to include scenarios where the user may send follow-up messages and interact with the agent across multiple turns. In addition, the codebase seems to contain logic for checking whether the environment should proceed to the next user turn.

However, when I exported HF traces using the oracle setting, I did not observe any cases where the user sends a follow-up message after the initial user request.

Export Setting

are-benchmark run \

  --hf-dataset meta-agents-research-environments/gaia2 \

  --hf-config mini \

  --hf-split validation \

  --oracle \

  --limit 100

Observed behavior

I exported 100 HF traces from the mini configuration with --oracle, but none of the traces appeared to contain user multi-turn interactions. In all inspected traces, the interaction seems to start from a single initial user request, and no subsequent user messages are triggered during the scenario execution.

Expected behavior

Based on the paper description and the code logic for detecting whether to enter the next user turn, I expected at least some scenarios to include user follow-up messages or multi-turn user-agent interactions.

Question

Could you clarify whether the current Gaia2 HF scenarios actually include user multi-turn interaction cases?

More specifically:

1. Are user multi-turn interactions included in the released HF dataset/configurations?

2. If so, which --hf-config or split should be used to observe them?

3. Does --oracle affect whether follow-up user turns are triggered?

4. Is the absence of user multi-turn interaction in the first 100 mini/validation traces expected behavior, or could this indicate an issue in the HF-packaged environment or runner logic?

Any guidance would be greatly appreciated. Thank you!

Environment

• ARE version: 1.2.0

• Python: 3.12