Skip to content

feat(robocasa_dc): cross-embodiment benchmark + reflex-dual model servers - #32

Open
MilkClouds wants to merge 4 commits into
mainfrom
feat/robocasa-dc-benchmark
Open

feat(robocasa_dc): cross-embodiment benchmark + reflex-dual model servers#32
MilkClouds wants to merge 4 commits into
mainfrom
feat/robocasa-dc-benchmark

Conversation

@MilkClouds

Copy link
Copy Markdown
Member

What

Adds the RoboCasa-DC cross-embodiment benchmark (SeeTraceAct, arXiv:2606.02745) and its in-process reflex model servers.

  • Benchmark (benchmarks/robocasa_dc): 24 RoboCasa kitchen tasks, cross-embodiment (GR-1 humanoid demo conditions a PandaOmron rollout). Scene restore is byte-identical to SeeTraceAct's validated pipeline — its restore wrapper + eval.py success criteria are imported as a module, not shelled out. Obs (53-d state + 3 flipped cameras) are pre-transformed exactly as SeeTraceAct's prepare_single_observation.
  • Model servers (in-process PredictModelServer, batched + sharded GPU inference):
    • reflex_dual_robocasa_dc — dual system (S2 cognition + GR00T-N1.5 S1), GR-1 demo conditioning, per-episode cognition cache, cognition_off ablation.
    • reflex_dual_robocasa_dc_s1 — S1-only base policy (K=0 cognition).
      Both load the reflex model in-process (launched from the model repo's venv, which provides reflex) — GR00T from reflex.models.gr00t_n15, the S1-only model from reflex.models.s1.
  • docker/Dockerfile.robocasa_dc for the osmesa sim deps.

Notes

  • Config paths are relative / env-overridable placeholders (${oc.env:ROBOCASA_DC_EVAL_*}, /path/to/...) — no cluster-specific paths.
  • The servers import reflex (the model repo) by design; they are launched from that repo's venv, so they are not exercised by this repo's own CI (deferred imports keep module import clean).

🤖 Generated with Claude Code

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the RoboCasa-DC cross-embodiment evaluation benchmark, including configuration files, a Dockerfile, a benchmark implementation, and model servers for both the dual-VLA and S1-only base policy. The review feedback highlights several opportunities to improve code robustness and resource management, such as replacing assertions with explicit ValueError exceptions for runtime validation, checking directory existence before listing contents, and using with open(...) context managers to prevent file descriptor leaks when loading configuration files.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

v = np.asarray(obs[key], dtype=np.float32).reshape(-1)
parts.append(v)
state = np.concatenate(parts)
assert state.shape[0] == STATE_DIM, f"assembled state {state.shape[0]}d, expected {STATE_DIM}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using assert statements for runtime data validation is discouraged because they can be globally disabled in Python when run with optimization flags (e.g., python -O). It is safer to raise a ValueError instead.

Suggested change
assert state.shape[0] == STATE_DIM, f"assembled state {state.shape[0]}d, expected {STATE_DIM}"
if state.shape[0] != STATE_DIM:
raise ValueError(f"assembled state {state.shape[0]}d, expected {STATE_DIM}")

Comment on lines +249 to +252
base = os.path.join(self.predefined_envs_root, task_name)
seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d)))
if not seeds:
raise FileNotFoundError(f"no seed dirs under {base}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the base directory does not exist, os.listdir(base) will raise a FileNotFoundError before the custom check if not seeds: is reached. It is cleaner and more robust to check if the directory exists first and raise a descriptive error.

Suggested change
base = os.path.join(self.predefined_envs_root, task_name)
seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d)))
if not seeds:
raise FileNotFoundError(f"no seed dirs under {base}")
base = os.path.join(self.predefined_envs_root, task_name)
if not os.path.isdir(base):
raise FileNotFoundError(f"Task directory not found: {base}")
seeds = sorted(int(d) for d in os.listdir(base) if d.isdigit() and os.path.isdir(os.path.join(base, d)))
if not seeds:
raise FileNotFoundError(f"no seed dirs under {base}")

model.to(device, torch.bfloat16).eval()
self.model = model

self.norm_stats = NormStats.from_dict(json.load(open(os.path.join(ckpt_dir, "norm_stats.json"))))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The file opened via open(...) is not explicitly closed, which can lead to resource leaks. Use a with open(...) context manager to ensure the file descriptor is properly released.

        with open(os.path.join(ckpt_dir, "norm_stats.json")) as f:
            self.norm_stats = NormStats.from_dict(json.load(f))

model.to(device, torch.bfloat16).eval()
self.model = model

self.norm_stats = NormStats.from_dict(json.load(open(os.path.join(ckpt_dir, "norm_stats.json"))))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The file opened via open(...) is not explicitly closed, which can lead to resource leaks. Use a with open(...) context manager to ensure the file descriptor is properly released.

        with open(os.path.join(ckpt_dir, "norm_stats.json")) as f:
            self.norm_stats = NormStats.from_dict(json.load(f))

@MilkClouds
MilkClouds force-pushed the feat/robocasa-dc-benchmark branch from ef5acf4 to b9adf1b Compare June 22, 2026 05:05
MilkClouds and others added 4 commits June 23, 2026 11:45
sync: allenai → origin (actions/checkout 6→7, allenai#76)
…vers

RoboCasa-DC (SeeTraceAct, arXiv:2606.02745) cross-embodiment eval: a GR-1 humanoid
demo conditions a Panda-arm rollout, restored byte-identically from predefined
envs via SeeTraceAct's wrapper (imported, not shelled out). In-process model
servers run the policy with batched/sharded GPU inference:
- reflex_dual_robocasa_dc: dual (S2 cognition + GR00T-N1.5 S1), demo conditioning,
  per-episode cognition cache, cognition on/off ablation.
- reflex_dual_robocasa_dc_s1: S1-only base policy (K=0 cognition).
Both load the reflex model in-process (launched from the model repo's venv) and
import GR00T from models.gr00t_n15 / the S1-only model from models.s1.
Config paths are placeholders/env-overridable; no cluster-specific paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RoboCasaDCBenchmark now captures a per-episode rollout video. `_extract_frame`
renders `robot0_agentview_center` (RoboCasa's own eval video camera — center-framed,
keeps the whole arm+task in view) at 512 directly from the sim, independent of the
policy obs cameras (which stay robot-mounted / low-res), and falls back to a policy
cam if the center camera is unavailable. Wired into reset()/step() via the
EpisodeRecorder. Adds gating_cat_rec.yaml (record_video=true) as an example.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hmarks)

RoboCasaDCBenchmark was the only StepBenchmark that fed the recorder video but
never called record_step, so step_rows was always empty. Add the standard
per-step logging: `_ALL_RECORD_FIELDS` + a `record_step(...)` in `step()`
capturing action(list)/eef_pos/gripper/reward/done/success, gated by the usual
`recording.record_step` config (off by default). Enables offline rollout
diagnostics (action jerk, EEF trajectory, grasp catch-vs-air) without changing
default behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant