| smoke_config | eval.yaml |
|---|
Multi-modal evaluation for robotic manipulation (ManiSkill3 fork / SAPIEN). Paper | GitHub
Docker image: ghcr.io/allenai/vla-evaluation-harness/robomme:latest
These configs default to render: cpu (lavapipe, shipped in the image): the native
NVIDIA path is ~5-10x faster but hangs at the first capture on some hosts, and no
startup probe can certify a host (issue #112). Pass --render gpu on a known-good
host; if Vulkan then cannot find the NVIDIA driver, uncomment the host ICD mount in
the config. Details: docs/render-backends.md.
| File | Description | Suites |
|---|---|---|
eval.yaml |
All 4 suites combined | 4 |
counting.yaml |
Counting suite only | 1 |
imitation.yaml |
Imitation suite only | 1 |
permanence.yaml |
Permanence suite only | 1 |
reference.yaml |
Reference suite only | 1 |