feat(robocasa_dc): cross-embodiment benchmark + reflex-dual model servers - #32
feat(robocasa_dc): cross-embodiment benchmark + reflex-dual model servers#32MilkClouds wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the RoboCasa-DC cross-embodiment evaluation benchmark, including configuration files, a Dockerfile, a benchmark implementation, and model servers for both the dual-VLA and S1-only base policy. The review feedback highlights several opportunities to improve code robustness and resource management, such as replacing assertions with explicit ValueError exceptions for runtime validation, checking directory existence before listing contents, and using with open(...) context managers to prevent file descriptor leaks when loading configuration files.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| v = np.asarray(obs[key], dtype=np.float32).reshape(-1) | ||
| parts.append(v) | ||
| state = np.concatenate(parts) | ||
| assert state.shape[0] == STATE_DIM, f"assembled state {state.shape[0]}d, expected {STATE_DIM}" |
There was a problem hiding this comment.
Using assert statements for runtime data validation is discouraged because they can be globally disabled in Python when run with optimization flags (e.g., python -O). It is safer to raise a ValueError instead.
| assert state.shape[0] == STATE_DIM, f"assembled state {state.shape[0]}d, expected {STATE_DIM}" | |
| if state.shape[0] != STATE_DIM: | |
| raise ValueError(f"assembled state {state.shape[0]}d, expected {STATE_DIM}") |
| base = os.path.join(self.predefined_envs_root, task_name) | ||
| seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d))) | ||
| if not seeds: | ||
| raise FileNotFoundError(f"no seed dirs under {base}") |
There was a problem hiding this comment.
If the base directory does not exist, os.listdir(base) will raise a FileNotFoundError before the custom check if not seeds: is reached. It is cleaner and more robust to check if the directory exists first and raise a descriptive error.
| base = os.path.join(self.predefined_envs_root, task_name) | |
| seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d))) | |
| if not seeds: | |
| raise FileNotFoundError(f"no seed dirs under {base}") | |
| base = os.path.join(self.predefined_envs_root, task_name) | |
| if not os.path.isdir(base): | |
| raise FileNotFoundError(f"Task directory not found: {base}") | |
| seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d))) | |
| if not seeds: | |
| raise FileNotFoundError(f"no seed dirs under {base}") |
| model.to(device, torch.bfloat16).eval() | ||
| self.model = model | ||
|
|
||
| self.norm_stats = NormStats.from_dict(json.load(open(os.path.join(ckpt_dir, "norm_stats.json")))) |
There was a problem hiding this comment.
| model.to(device, torch.bfloat16).eval() | ||
| self.model = model | ||
|
|
||
| self.norm_stats = NormStats.from_dict(json.load(open(os.path.join(ckpt_dir, "norm_stats.json")))) |
There was a problem hiding this comment.
ef5acf4 to
b9adf1b
Compare
sync: allenai → origin (actions/checkout 6→7, allenai#76)
…vers RoboCasa-DC (SeeTraceAct, arXiv:2606.02745) cross-embodiment eval: a GR-1 humanoid demo conditions a Panda-arm rollout, restored byte-identically from predefined envs via SeeTraceAct's wrapper (imported, not shelled out). In-process model servers run the policy with batched/sharded GPU inference: - reflex_dual_robocasa_dc: dual (S2 cognition + GR00T-N1.5 S1), demo conditioning, per-episode cognition cache, cognition on/off ablation. - reflex_dual_robocasa_dc_s1: S1-only base policy (K=0 cognition). Both load the reflex model in-process (launched from the model repo's venv) and import GR00T from models.gr00t_n15 / the S1-only model from models.s1. Config paths are placeholders/env-overridable; no cluster-specific paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RoboCasaDCBenchmark now captures a per-episode rollout video. `_extract_frame` renders `robot0_agentview_center` (RoboCasa's own eval video camera — center-framed, keeps the whole arm+task in view) at 512 directly from the sim, independent of the policy obs cameras (which stay robot-mounted / low-res), and falls back to a policy cam if the center camera is unavailable. Wired into reset()/step() via the EpisodeRecorder. Adds gating_cat_rec.yaml (record_video=true) as an example. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hmarks) RoboCasaDCBenchmark was the only StepBenchmark that fed the recorder video but never called record_step, so step_rows was always empty. Add the standard per-step logging: `_ALL_RECORD_FIELDS` + a `record_step(...)` in `step()` capturing action(list)/eef_pos/gripper/reward/done/success, gated by the usual `recording.record_step` config (off by default). Enables offline rollout diagnostics (action jerk, EEF trajectory, grasp catch-vs-air) without changing default behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
b9adf1b to
b7110b4
Compare
What
Adds the RoboCasa-DC cross-embodiment benchmark (SeeTraceAct, arXiv:2606.02745) and its in-process reflex model servers.
benchmarks/robocasa_dc): 24 RoboCasa kitchen tasks, cross-embodiment (GR-1 humanoid demo conditions a PandaOmron rollout). Scene restore is byte-identical to SeeTraceAct's validated pipeline — its restore wrapper +eval.pysuccess criteria are imported as a module, not shelled out. Obs (53-d state + 3 flipped cameras) are pre-transformed exactly as SeeTraceAct'sprepare_single_observation.PredictModelServer, batched + sharded GPU inference):reflex_dual_robocasa_dc— dual system (S2 cognition + GR00T-N1.5 S1), GR-1 demo conditioning, per-episode cognition cache,cognition_offablation.reflex_dual_robocasa_dc_s1— S1-only base policy (K=0 cognition).Both load the reflex model in-process (launched from the model repo's venv, which provides
reflex) — GR00T fromreflex.models.gr00t_n15, the S1-only model fromreflex.models.s1.docker/Dockerfile.robocasa_dcfor the osmesa sim deps.Notes
${oc.env:ROBOCASA_DC_EVAL_*},/path/to/...) — no cluster-specific paths.import reflex(the model repo) by design; they are launched from that repo's venv, so they are not exercised by this repo's own CI (deferred imports keep module import clean).🤖 Generated with Claude Code