| smoke_config |
|---|
Large-scale household activity benchmark (OmniGibson/Isaac Sim). Paper | GitHub
Docker image: ghcr.io/allenai/vla-evaluation-harness/behavior1k:latest
Protocol version: targets the BEHAVIOR Challenge 2025 stack (v3.7.2, Isaac Sim
4.5.0, B50). It does not exercise the official 2026 evaluator (v3.9.1, 100 tasks,
python -m omnigibson.eval.eval); see issue #113.
Requires an R1Pro-compatible model server. See behavior1k.md.
| File | Description | Tasks | Episodes/task |
|---|---|---|---|
eval.yaml |
Full BEHAVIOR-1K evaluation | 50 | 5 |