configs

Evaluation Configs

Quick reference for running evaluations. For reproduction scores, see docs/reproductions/.

Quick Start

# 1. Start a model server (terminal 1)
vla-eval serve --config configs/model_servers/xvla/libero.yaml

# 2. Run an evaluation (terminal 2)
vla-eval run --config configs/benchmarks/libero/all.yaml

Benchmarks

Benchmark names link to their config directory with available YAML files and usage details.

Benchmark		Paper	Docker Image	Python	Description
LIBERO		2310.07899	libero	3.8	Tabletop manipulation, 4 suites (MuJoCo)
LIBERO-Pro		2310.07899	libero-pro	3.8	Extended harder tasks
LIBERO-Plus		2310.07899	libero-plus	3.8	Extended task set
LIBERO-Mem		2310.07899	libero-mem	3.8	Memory-augmented tasks
SimplerEnv		2405.05941	simpler	3.10	Google Robot + WidowX real2sim (SAPIEN)
CALVIN		2112.03227	calvin	3.8	Chained 5-subtask sequences (PyBullet)
RoboTwin		2409.02920	robotwin	3.10	Dual-arm manipulation (SAPIEN)
DuoBench		2606.11901	duobench	3.11	Bimanual manipulation, Franka FR3 Duo (MuJoCo)
ManiSkill2		2302.04659	maniskill2	3.10	Generalizable manipulation (SAPIEN)
RoboMME		2603.04639	robomme	3.11	Multi-modal evaluation (SAPIEN)
Kinetix		2410.23208	kinetix	3.11	Physics-based 2D manipulation (JAX)
MolmoSpaces-Bench		2603.16861	molmospaces	3.11	Spatial reasoning (AI2-THOR)
RLBench		1909.12271	🔒 rlbench	3.8	Vision-guided manipulation (CoppeliaSim)
RoboCasa		2406.02523	robocasa	3.10	Kitchen manipulation, 24 atomic tasks (MuJoCo)
RoboCasa365		2603.04356	robocasa365	3.11	Multi-task kitchen manipulation, 50 target tasks (MuJoCo)
VLABench		2502.09858	vlabench	3.10	Language-conditioned manipulation (MuJoCo)
MIKASA-Robo		2502.07007	mikasa-robo	3.10	Robot manipulation (SAPIEN)
RoboCerebra		2502.02853	robocerebra	3.8	Cognitive manipulation (MuJoCo)
BEHAVIOR-1K		2403.09227	🔒 behavior1k	3.10	Household activities (OmniGibson)
RoboDojo		2607.04434	🔒 robodojo	3.11	Bimanual dual ARX-X5, 42 tasks (Isaac Lab)

_{reproduced · integrated, awaiting first reproduction · 🔒 license-restricted (local build only)}

Model Servers

Model names link to their server config directory. For reproduction scores, see docs/reproductions/.

Model		Paper	Codebase	Supported Benchmarks	Reproduction
OpenVLA		2406.09246	openvla/openvla	LIBERO, SimplerEnv GR	report
π₀ / π₀-FAST		2410.24164	Physical-Intelligence/openpi	LIBERO	report
GR00T N1.6		2503.14734	NVIDIA/Isaac-GR00T	LIBERO, SimplerEnv	report
OFT		2502.19645	moojink/openvla-oft	LIBERO	report
X-VLA		2510.10274	2toinf/X-VLA	LIBERO, CALVIN, SimplerEnv, RoboTwin	report
CogACT		2411.19650	microsoft/CogACT	SimplerEnv	report
VLANeXt		2602.18532	DravenALG/VLANeXt	LIBERO	PR #34
DB-CogACT		2510.23511	Dexmal/dexbotic	LIBERO, CALVIN, SimplerEnv, RoboTwin, ManiSkill2	report
starVLA		2604.05014	starVLA/starVLA	LIBERO, SimplerEnv	report
RTC		2506.07339	Physical-Intelligence/rtc	Kinetix	report
MolmoBot		2603.16861	allenai/MolmoBot	MolmoSpaces	report
MME-VLA		2603.04639	RoboMME/robomme_policy_learning	RoboMME	report
π₀.₅ (RoboDojo)		2607.04434	XPolicyLab/XPolicyLab	RoboDojo	report

_{reproduced · integrated, awaiting first reproduction}

Config Schemas

Benchmark configs (benchmarks/<name>/*.yaml)

Used by vla-eval run.

server:
  url: "ws://localhost:8000"
docker:
  image: ghcr.io/allenai/vla-evaluation-harness/<name>:latest
output_dir: "./results"
render: gpu                  # optional: gpu (default) | cpu — see --render
benchmarks:
  - benchmark: "vla_eval.benchmarks.<name>.benchmark:ClassName"
    episodes_per_task: 50
    params: { ... }

render is run-level rather than per entry: the renderer binds at the first simulator import. cpu software-renders and starts the container with no GPU attached; benchmarks that don't declare CPU support are rejected before the run starts. See Render Backends.

Server configs (model_servers/<name>/*.yaml)

Used by vla-eval serve.

extends: _base.yaml          # optional inheritance
script: "src/vla_eval/model_servers/mymodel.py"
args:
  model_path: org/model-name
  chunk_size: 16

The extends mechanism deep-merges args from a base file; script is inherited if omitted. Model servers declare their observation requirements via the HELLO handshake, so the benchmark is auto-configured without manual flags.