Add official-protocol RoboCasa365 GR00T evaluation

configs/README.md

@@ -31,7 +31,7 @@ Benchmark names link to their config directory with available YAML files and usa
 | [Kinetix](benchmarks/kinetix/) | ![◇](https://img.shields.io/badge/◇-blue) | [2410.23208](https://arxiv.org/abs/2410.23208) | [`kinetix`](https://ghcr.io/allenai/vla-evaluation-harness/kinetix) | 3.11 | Physics-based 2D manipulation (JAX) |
 | [MolmoSpaces-Bench](benchmarks/molmospaces/) | ![✓](https://img.shields.io/badge/✓-teal) | [2603.16861](https://arxiv.org/abs/2603.16861) | [`molmospaces`](https://ghcr.io/allenai/vla-evaluation-harness/molmospaces) | 3.11 | Spatial reasoning (AI2-THOR) |
 | [RLBench](benchmarks/rlbench/) | ![◇](https://img.shields.io/badge/◇-blue) | [1909.12271](https://arxiv.org/abs/1909.12271) | 🔒 `rlbench` | 3.8 | Vision-guided manipulation (CoppeliaSim) |
-| [RoboCasa](benchmarks/robocasa/) | ![◇](https://img.shields.io/badge/◇-blue) | [2406.02523](https://arxiv.org/abs/2406.02523) | [`robocasa`](https://ghcr.io/allenai/vla-evaluation-harness/robocasa) | 3.11 | Kitchen manipulation (MuJoCo) |
+| [RoboCasa365](benchmarks/robocasa/) | ![◇](https://img.shields.io/badge/◇-blue) | [2603.04356](https://arxiv.org/abs/2603.04356) | [`robocasa`](https://ghcr.io/allenai/vla-evaluation-harness/robocasa) | 3.11 | 50-task kitchen manipulation (MuJoCo) |
 | [VLABench](benchmarks/vlabench/) | ![◇](https://img.shields.io/badge/◇-blue) | [2502.09858](https://arxiv.org/abs/2502.09858) | [`vlabench`](https://ghcr.io/allenai/vla-evaluation-harness/vlabench) | 3.10 | Language-conditioned manipulation (SAPIEN) |
 | [MIKASA-Robo](benchmarks/mikasa/) | ![◇](https://img.shields.io/badge/◇-blue) | [2502.07007](https://arxiv.org/abs/2502.07007) | [`mikasa-robo`](https://ghcr.io/allenai/vla-evaluation-harness/mikasa-robo) | 3.10 | Robot manipulation (MuJoCo) |
 | [RoboCerebra](benchmarks/robocerebra/) | ![◇](https://img.shields.io/badge/◇-blue) | [2502.02853](https://arxiv.org/abs/2502.02853) | [`robocerebra`](https://ghcr.io/allenai/vla-evaluation-harness/robocerebra) | 3.8 | Cognitive manipulation (MuJoCo) |

configs/benchmarks/robocasa/README.md

@@ -1,16 +1,136 @@
 ---
-smoke_config: eval.yaml
+smoke_config: smoke.yaml
 ---
 
-# RoboCasa
+# RoboCasa365
 
-Kitchen manipulation benchmark (MuJoCo/robosuite).
-[Paper](https://arxiv.org/abs/2406.02523) | [GitHub](https://github.com/robocasa/robocasa)
+Official 50-task multi-task benchmark using RoboCasa's Panda-Omron Gymnasium wrapper.
+[Paper](https://robocasa.ai/assets/robocasa365_iclr26.pdf) ·
+[Code](https://github.com/robocasa/robocasa) ·
+[Protocol](https://robocasa.ai/docs/build/html/benchmarking/multitask_learning.html) ·
+[Leaderboard](https://robocasa.ai/leaderboard.html)
 
 **Docker image:** `ghcr.io/allenai/vla-evaluation-harness/robocasa:latest`
 
 ## Configs
 
 | File | Description | Tasks | Episodes/task |
-|------|-------------|:-----:|:-------------:|
-| `eval.yaml` | Full RoboCasa evaluation | 24 | 50 |
+| --- | --- | ---: | ---: |
+| `rc365.yaml` | Official multi-task protocol | 50 | 50 |
+| `rc365_s2b_qualification.yaml` | Base config for one S2B qualification shard | 1 | 1 |
+| `smoke.yaml` | Contract smoke test | 2 | 1 |
+
+The adapter reads task membership and task-specific horizons from RoboCasa's registry.
+It evaluates the target50 tasks in pretraining kitchens (`split: pretrain`), matching the official multi-task leaderboard protocol.
+Success is sampled and accumulated at the end of each 16-action chunk, while every episode continues to its task-specific horizon.
+The wire contract preserves all 12 Panda-Omron dimensions in the official dataset order instead of padding a 7-D arm action.
+
+## Qualification-slice parity
+
+The parity script compares the isolated RC365 adapter with the read-only
+`rc365_s2b.environments.OfficialRoboCasaEnv` for the six System 1
+qualification tasks at seed 0. It checks repeated-reset determinism, raw
+observation keys, shapes, dtypes, values, and simulator state across the same
+recorded 32-step action sequence.
+
+```bash
+SIM_PYTHON=/path/to/system1/.venv/bin/python
+REFERENCE_SRC=/path/to/projects/rc365-s2b/src
+MUJOCO_GL=egl NUMBA_DISABLE_JIT=1 "$SIM_PYTHON" \
+  scripts/check_rc365_adapter_parity.py \
+  --reference-src "$REFERENCE_SRC"
+```
+
+The output is `results/rc365_adapter_qualification_parity.json`. The script
+redirects RoboCasa's generated temporary XML files to `/tmp`; it does not
+modify the simulator environment or the reference source tree. Full GR00T or
+MLLM inference is outside adapter parity and remains GPU or Slurm-deferred.
+
+## Implementation boundary
+
+- Reused upstream: the registered RoboCasa Gym environment, task registry, pretrain split, per-task horizon, observation schema, and strict success predicate.
+- Implemented here: canonical observation mapping and lossless 12-D named-action decoding.
+- Dependency boundary: Docker verifies RoboCasa `1.0.1` and robosuite `1.5.2` at tested patch revisions because those upstream fixes have no new semantic tags.
+
+## CPU rendering
+
+Set `VLA_EVAL_RENDER=cpu` to select Mesa llvmpipe through OSMesa. Set
+`docker.gpus: none`, or pass `--gpus none`, so the benchmark container receives
+no GPU device. The model server remains on the host GPU. GPU rendering through
+the EGL device platform remains the default.
+
+```bash
+VLA_EVAL_RENDER=cpu vla-eval run \
+  --config configs/benchmarks/robocasa/smoke.yaml \
+  --gpus none
+```
+
+CPU mode sets `MUJOCO_GL=osmesa` and `PYOPENGL_PLATFORM=osmesa`, forces
+software GL, and clears `EGL_PLATFORM` and `MUJOCO_EGL_DEVICE_ID` before
+MuJoCo is imported.
+
+Container verification requires a Docker host:
+
+```bash
+docker build -f docker/Dockerfile.base -t vla-eval-base:rc365-cpu .
+docker build \
+  --build-arg BASE_IMAGE=vla-eval-base:rc365-cpu \
+  -f docker/Dockerfile.robocasa \
+  -t vla-eval-robocasa:rc365-cpu .
+docker run --rm \
+  -e VLA_EVAL_RENDER=cpu \
+  -e NVIDIA_VISIBLE_DEVICES=void \
+  -e CUDA_VISIBLE_DEVICES=-1 \
+  --entrypoint /opt/conda/bin/conda \
+  vla-eval-robocasa:rc365-cpu \
+  run --no-capture-output -n robocasa \
+  python /workspace/scripts/check_robocasa_cpu_render.py
+```
+
+## RC365 S2B qualification
+
+The qualification runner expands the reference seed manifest in stable task
+and seed order, then interleaves `gold-s2`, `global-s1`, and `random-valid`.
+Each invocation owns exactly one task, seed, and condition. Qualification-only
+success checks and 16-step chunk telemetry live in
+`RoboCasaS2BQualificationBenchmark`, leaving the upstream adapters unchanged.
+
+```bash
+export RC365_S2B_ROOT=/path/to/projects/rc365-s2b
+export RC365_PHASE_MANIFEST=/path/to/phase_manifest.jsonl
+export ROBOCASA_GR00T_N15_CKPT=/path/to/checkpoint-120000
+export ROBOCASA_MODALITY_JSON=/path/to/modality.json
+export VLA_EVAL_ROBOCASA_IMAGE=vla-eval-robocasa:rc365-cpu
+
+uv run python -m vla_eval.rc365_s2b_qualification resolve \
+  --rung dev --array-index 0 \
+  --seed-manifest "$RC365_S2B_ROOT/config/qualification_seeds.json"
+
+uv run python -m vla_eval.rc365_s2b_qualification run \
+  --rung dev --array-index 0 \
+  --seed-manifest "$RC365_S2B_ROOT/config/qualification_seeds.json" \
+  --phase-manifest "$RC365_PHASE_MANIFEST" \
+  --render cpu --dev
+```
+
+Outputs are one-record JSONL files under
+`results/rc365_s2b_qualification/<rung>/<condition>/`. Records use schema
+`rc365-s2b-exec-episode-v1` and include strict success, termination, steps,
+chunks, System 2 calls, run configuration, and provenance.
+
+```bash
+rc365-s2b score-qualification \
+  results/rc365_s2b_qualification/dev/gold-s2 \
+  results/rc365_s2b_qualification/dev/global-s1 \
+  results/rc365_s2b_qualification/dev/random-valid
+```
+
+Submit the Slurm template with the rung's array bound. CPU rendering requests
+one GPU for the policy. GPU rendering needs two GPUs.
+
+```bash
+# dev: 0-17, training-dev: 0-89, frozen: 0-449
+sbatch --array=0-17 \
+  --export=ALL,QUALIFICATION_RUNG=dev,VLA_EVAL_RENDER=cpu,VLA_EVAL_ROBOCASA_IMAGE=vla-eval-robocasa:rc365-cpu \
+  scripts/run_rc365_s2b_qualification.sbatch
+```

configs/benchmarks/robocasa/eval.yaml

@@ -1,14 +1,11 @@
-# RoboCasa kitchen manipulation evaluation (6 tasks × 5 episodes)
-#
-# RoboCasa provides 365 kitchen tasks on 2500+ procedurally-generated scenes.
-# Default config uses a small subset of atomic tasks for quick evaluation.
-#
-# Requires ~10GB of kitchen assets from HuggingFace.
+# RoboCasa kitchen manipulation, default atomic subset (6 tasks x 5 episodes).
+# Needs ~10GB of kitchen assets from HuggingFace.
 server:
   url: "ws://localhost:8000"
 
 docker:
   image: ghcr.io/allenai/vla-evaluation-harness/robocasa:latest
+  env: [VLA_EVAL_RENDER]
 
 output_dir: "./results"
 
@@ -21,4 +18,3 @@ benchmarks:
       max_steps: 500
       split: pretrain
       seed: null
-

configs/benchmarks/robocasa/rc365.yaml

@@ -0,0 +1,19 @@
+# RoboCasa365 official multi-task evaluation: 50 tasks x 50 seeded episodes.
+server:
+  url: "ws://localhost:8000"
+
+docker:
+  image: ghcr.io/allenai/vla-evaluation-harness/robocasa:latest
+  env: [VLA_EVAL_RENDER]
+
+output_dir: "./results"
+
+benchmarks:
+  - benchmark: "vla_eval.benchmarks.robocasa.rc365:RoboCasa365Benchmark"
+    episodes_per_task: 50
+    params:
+      split: pretrain
+      seed: 0
+      camera_size: 256
+      enable_render: true
+      success_check_interval: 16

configs/benchmarks/robocasa/smoke.yaml

@@ -0,0 +1,21 @@
+# Short contract smoke test; this is not a leaderboard protocol.
+server:
+  url: "ws://localhost:8000"
+
+docker:
+  image: ghcr.io/allenai/vla-evaluation-harness/robocasa:latest
+  env: [VLA_EVAL_RENDER]
+
+output_dir: "./results"
+
+benchmarks:
+  - benchmark: "vla_eval.benchmarks.robocasa.rc365:RoboCasa365Benchmark"
+    episodes_per_task: 1
+    max_steps: 5
+    params:
+      tasks: [KettleBoiling, ArrangeBreadBasket]
+      split: pretrain
+      seed: 0
+      camera_size: 256
+      enable_render: true
+      success_check_interval: 16

configs/model_servers/groot/README.md

@@ -2,7 +2,7 @@
 smoke_config: groot.yaml
 ---
 
-# GR00T N1.6
+# GR00T
 
 3B dual-system VLA from NVIDIA. [Paper](https://arxiv.org/abs/2503.14734) | [GitHub](https://github.com/NVIDIA/Isaac-GR00T)
 
@@ -14,3 +14,8 @@ smoke_config: groot.yaml
 | `libero.yaml` | LIBERO | `nvidia/GR00T-N1.6-3B` |
 | `simpler_widowx.yaml` | SimplerEnv WidowX | `nvidia/GR00T-N1.6-bridge` |
 | `simpler_google_robot.yaml` | SimplerEnv GR | `nvidia/GR00T-N1.6-fractal` |
+| `robocasa_n15.yaml` | RoboCasa365 | `robocasa/robocasa365_checkpoints` |
+
+For RoboCasa365, set `ROBOCASA_GR00T_N15_CKPT` to `gr00t_n1-5/multitask_learning/checkpoint-120000` from `robocasa/robocasa365_checkpoints`.
+The pinned RoboCasa GR00T fork directly imports FlashAttention, so its script pins the exercised FlashAttention build instead of using the Transformers `kernels` extra.
+The configured seed fixes the diffusion-noise sequence.

configs/model_servers/groot/robocasa_n15.yaml

@@ -0,0 +1,10 @@
+# RoboCasa365 GR00T N1.5 multitask checkpoint.
+# Download robocasa/robocasa365_checkpoints and point the environment variable
+# at gr00t_n1-5/multitask_learning/checkpoint-120000.
+script: "src/vla_eval/model_servers/robocasa_groot.py"
+args:
+  model_path: "${oc.env:ROBOCASA_GR00T_N15_CKPT,${oc.env:VLA_EVAL_ASSETS_CACHE,${oc.env:XDG_CACHE_HOME,${oc.env:HOME}/.cache}/vla-eval/assets}/robocasa/gr00t_n1-5/multitask_learning/checkpoint-120000}"
+  chunk_size: 16
+  denoising_steps: 4
+  seed: 0
+  port: 8000

configs/model_servers/rc365_s2b/README.md

@@ -0,0 +1,12 @@
+# RC365 S2B hierarchical policy
+
+One stateful policy wrapping the S2B System 2 planner and the GR00T
+System 1 executor. System 2 is re-queried after every 16-step action
+chunk: an unchanged call continues the current instruction, a new call
+switches it, and finish_task terminates the episode (surfaced to the
+runner as `terminate_episode`).
+
+`hierarchical.yaml` selects the System 2 implementation (`gold`,
+`global_only`, `random_valid`, or `mllm`) and the checkpoint paths.
+Set `VLA_EVAL_RENDER=cpu` to render through OSMesa and keep the GPU
+exclusively for the policy.

configs/model_servers/rc365_s2b/hierarchical.yaml

@@ -0,0 +1,11 @@
+# RC365 S2B hierarchical policy. The reference package must be on PYTHONPATH.
+script: "src/vla_eval/model_servers/rc365_s2b.py"
+args:
+  checkpoint: "${oc.env:ROBOCASA_GR00T_N15_CKPT,${oc.env:VLA_EVAL_ASSETS_CACHE,${oc.env:XDG_CACHE_HOME,${oc.env:HOME}/.cache}/vla-eval/assets}/robocasa/gr00t_n1-5/multitask_learning/checkpoint-120000}"
+  modality_path: "${oc.env:ROBOCASA_MODALITY_JSON,/path/to/modality.json}"
+  registry_path: "${oc.env:RC365_S2B_ROOT,/path/to/rc365-s2b}/config/registry_composite.json"
+  system2: global-only
+  device: cuda
+  denoising_steps: 4
+  seed: 0
+  port: 8000

docker/Dockerfile.base

@@ -1,6 +1,6 @@
 # Common base image for all VLA evaluation harness benchmarks
 #
-# Provides: CUDA 12.1 runtime, EGL/Vulkan rendering, Miniforge, uv
+# Provides: CUDA 12.1 runtime, NVIDIA or Mesa rendering, Miniforge, uv
 #
 FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04
 
@@ -18,8 +18,9 @@ ENV DEBIAN_FRONTEND=noninteractive \
 RUN apt-get update && apt-get install -y --no-install-recommends \
         build-essential cmake git wget curl ca-certificates unzip \
         # EGL / OpenGL
-        libegl1 libegl1-mesa-dev libgl1 libgl1-mesa-glx libgles2 \
+        libegl1 libegl-mesa0 libegl1-mesa-dev libgl1 libgl1-mesa-glx libglx-mesa0 libgles2 \
         libglfw3 libosmesa6 libosmesa6-dev libglew-dev libglvnd-dev \
+        mesa-utils \
         # Vulkan
         libvulkan1 mesa-vulkan-drivers \
         # X11 / misc

docker/Dockerfile.robocasa

@@ -7,24 +7,28 @@ RUN conda create -n robocasa python=3.11 pip -y && conda clean -afy
 
 SHELL ["conda", "run", "-n", "robocasa", "/bin/bash", "-c"]
 
-ARG ROBOCASA_ROBOSUITE_REF=85abee228d1c43ab1939bce33028099945d453b4
+ARG ROBOCASA_ROBOSUITE_VERSION=1.5.2
+ARG ROBOCASA_ROBOSUITE_REF=5ce6643f3092639d08f7b0f90ed1c6a84f50552c
 RUN mkdir -p /app/robosuite && \
     cd /app/robosuite && \
     git init -q && \
     git remote add origin https://github.com/ARISE-Initiative/robosuite.git && \
     git fetch -q --depth 1 origin "${ROBOCASA_ROBOSUITE_REF}" && \
     git checkout -q FETCH_HEAD && \
     uv pip install --no-cache-dir -e . && \
+    test "$(python -c 'from importlib.metadata import version; print(version("robosuite"))')" = "${ROBOCASA_ROBOSUITE_VERSION}" && \
     rm -rf /app/robosuite/.git
 
-ARG ROBOCASA_REF=v1.0
+ARG ROBOCASA_VERSION=1.0.1
+ARG ROBOCASA_REF=b4684e6ee37d377cc392e98302a6b916d588b415
 RUN mkdir -p /app/robocasa && \
     cd /app/robocasa && \
     git init -q && \
     git remote add origin https://github.com/robocasa/robocasa.git && \
     git fetch -q --depth 1 origin "${ROBOCASA_REF}" && \
     git checkout -q FETCH_HEAD && \
     uv pip install --no-cache-dir -e . && \
+    test "$(python -c 'from importlib.metadata import version; print(version("robocasa"))')" = "${ROBOCASA_VERSION}" && \
     rm -rf /app/robocasa/.git
 
 # Setup macros (non-interactive) and download kitchen assets (~10 GB)
@@ -39,6 +43,7 @@ ARG HARNESS_VERSION=0.0.0
 ENV SETUPTOOLS_SCM_PRETEND_VERSION=${HARNESS_VERSION}
 RUN uv pip install --no-cache-dir -e .
 COPY configs/ configs/
+COPY scripts/check_robocasa_cpu_render.py scripts/
 
 ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "robocasa", "vla-eval"]
-CMD ["run", "--config", "/workspace/configs/robocasa_eval.yaml"]
+CMD ["run", "--config", "/workspace/configs/benchmarks/robocasa/rc365.yaml"]

scripts/check_robocasa_cpu_render.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python3
+"""Create, step, and render the three RC365 cameras with Mesa llvmpipe."""
+
+from __future__ import annotations
+
+import json
+import os
+
+os.environ["VLA_EVAL_RENDER"] = "cpu"
+os.environ.setdefault("CUDA_VISIBLE_DEVICES", "-1")
+
+import numpy as np
+
+from vla_eval.benchmarks.robocasa.rc365 import ACTION_DIM, VIDEO_KEYS, RoboCasa365Benchmark
+from vla_eval.recording import NullEpisodeRecorder
+
+
+def main() -> None:
+    benchmark = RoboCasa365Benchmark(
+        tasks=["RinseSinkBasin"],
+        split="pretrain",
+        seed=0,
+        max_steps=16,
+    )
+    benchmark._recorder = NullEpisodeRecorder()
+    try:
+        task = {"name": "RinseSinkBasin", "episode_idx": 0}
+        raw = benchmark.reset(task)
+        first = benchmark.make_obs(raw, task)
+        stepped = benchmark.step({"actions": np.zeros(ACTION_DIM, dtype=np.float32)})
+        second = benchmark.make_obs(stepped.obs, task)
+        for observation in (first, second):
+            images = observation["images"]
+            missing = [key for key in VIDEO_KEYS if key not in images]
+            if missing:
+                raise RuntimeError(f"missing rendered cameras: {missing}")
+            for key in VIDEO_KEYS:
+                image = np.asarray(images[key])
+                if image.shape != (256, 256, 3) or image.dtype != np.uint8:
+                    raise RuntimeError(f"unexpected {key} image: shape={image.shape}, dtype={image.dtype}")
+        print(
+            json.dumps(
+                {
+                    "camera_shapes": {key: list(np.asarray(second["images"][key]).shape) for key in VIDEO_KEYS},
+                    "cuda_visible_devices": os.environ.get("CUDA_VISIBLE_DEVICES"),
+                    "egl_platform": os.environ.get("EGL_PLATFORM"),
+                    "mujoco_gl": os.environ.get("MUJOCO_GL"),
+                    "pyopengl_platform": os.environ.get("PYOPENGL_PLATFORM"),
+                    "pass": True,
+                },
+                sort_keys=True,
+            )
+        )
+    finally:
+        benchmark.cleanup()
+
+
+if __name__ == "__main__":
+    main()

src/vla_eval/benchmarks/robocasa/__init__.py

@@ -1,5 +1,6 @@
-"""RoboCasa kitchen manipulation benchmark."""
+"""RoboCasa and RoboCasa365 kitchen manipulation benchmarks."""
 
 from vla_eval.benchmarks.robocasa.benchmark import RoboCasaBenchmark
+from vla_eval.benchmarks.robocasa.rc365 import RoboCasa365Benchmark
 
-__all__ = ["RoboCasaBenchmark"]
+__all__ = ["RoboCasa365Benchmark", "RoboCasaBenchmark"]