~22% of AWM-1K environments have structural seed SQL errors

[wumengz]

Hi team, thanks for the great work! The idea of synthesizing executable environments at scale for agentic RL is very exciting, and the released dataset has been extremely useful for our research.

I'm running evaluations on the released AWM-1K environments and noticed that 225 out of ~1000 environments consistently produce server_error during verification, caused by seed SQL errors. These errors are 100% reproducible across multiple independent runs and replay.

Questions

1. Were these environments excluded during RL training? (The paper uses 526 environments — was this subset selected to avoid broken ones?)
2. Any plan to fix and re-release the dataset?

Thanks!

[Raibows]

Hi,
Thank you for your interests. Envs are random selected for training. However, through the whole training, as mentioned in our paper, we noticed very few ~4% server_error. Note that the judge outcome server_error is not only related to the env itself but also task, agent behavior, and judge model. And what do you mean by seed sql error? Do you mean the error exist in populating the seed data for the env, or in the verification code?

To help investigate this, could you please share more stats or logs? For example, are all tasks of that env server_error, are all rollouts of that env server_error, what agent model did you use, did you manually inspect the judge results?

[Jake-Song]

Thanks for your research and implement in advance.
Maybe seed sql error @wumengz said is the error of constructing DB with gen_sample.jsonl.
These are caused by incorrect escaping or malform values for INSERT SQL query

log like these

Failed to insert into workflow_step_runs: near "7": syntax error
Failed to insert into devices: table devices has no column named current_status
Failed to insert into service_stops: near "1": syntax error
Failed to insert into route_stops: UNIQUE constraint failed: route_stops.service_stop_id
Failed to insert into run_logs: near "2": syntax error