clawarena check --framework openclaw fails with many errors – maybe my setup is wrong?

[Lautisticyc]

Hi, I'm new to projects and this might be a very beginner mistake – please bear with me 🙏

I followed the README to set up the environment (pip install -e .[dev] etc.) and then ran:

clawarena check --data data/clawarena/tests.json --framework openclaw

But I got many errors like these (I only pasted the first few lines – the full list is long and attached below):

[ERROR] hil_c1: round 'q7': exec_check feedback.incorrect must be non-empty string
[ERROR] hil_c1: round 'q10': exec_check feedback.incorrect must be non-empty string
...
[ERROR] [openclaw] agent 'hil_c1' in manifest but not in openclaw.json
[ERROR] [openclaw] agent 'hil_c2' in manifest but not in openclaw.json
...
[ERROR] [openclaw] hil_c1/q5: update_id '2' not found in manifest.updates
...
[ERROR] [openclaw] hil_i1/update:upd4_sessions/chief_zhang_im_cb2db319-4dd5-4bf4-98ba-7f9ba04b3fb8.jsonl: consecutive 'assistant' messages at line 4
...

log.txt

I have installed openclaw globally and it works (I can run openclaw --version).
But I'm not sure if I missed some data preparation step – the README didn't mention extra download or generation of openclaw.json under data/clawarena/openclaw/.
I checked the data/clawarena/ folder and I do have tests.json, but it seems the referenced openclaw.json is missing(no agent definitions for hil_*).
Also some exec_check.feedback strings are empty, and many update_id values are not defined in manifest.updates.
And one more problem is the consecutive 'assistant' messages?

Could this be because the benchmark data is not fully committed to the repository? Or is there a separate data download / generation script that I missed?

If it's just my local misconfiguration, please point me in the right direction – any help would be greatly appreciated.

Environment

OS: WSL2 Ubuntu 24.04.1 LTS

Python 3.12.3

ClawArena commit: latest main (git clone today)

openclaw version: 2026.5.7 (eeef486)

Thank you!

[kevinx0522]

Thanks for your patience, and sorry for the long delay. We have been working hard on this project recently with massive updates to both datasets and source code. We have also re-run the leaderboard. Please check the latest version.