Skip to content

Add official-protocol RoboCasa365 GR00T evaluation#103

Open
MilkClouds wants to merge 21 commits into
mainfrom
rc365-s2b-adapter
Open

Add official-protocol RoboCasa365 GR00T evaluation#103
MilkClouds wants to merge 21 commits into
mainfrom
rc365-s2b-adapter

Conversation

@MilkClouds

@MilkClouds MilkClouds commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add an opt-in rc365 protocol for the official target50 registry, pretraining-kitchen split, per-task horizons, 16-step success checks, and lossless 12-D Panda-Omron actions
  • preserve the original arbitrary-task, configurable-camera, 7-D adapter and keep eval.yaml byte-for-byte unchanged
  • add full and smoke RC365 configs plus a seeded GR00T N1.5 server
  • pin the exercised RoboCasa and robosuite patch revisions while checking their semantic package versions

Scope

This PR changes only the RoboCasa adapter/config/docs, the GR00T server/config, and the RoboCasa image.
It contains no simulator-bound test, generated report, generic result plumbing, or checked-in parity experiment.

Dependency policy

  • the PEP 723 GR00T script uses exclude-newer = "2026-07-19T00:00:00Z"
  • the pinned fork's bundled Eagle model imports flash_attn directly, so Transformers kernels cannot replace it without changing upstream model code
  • the exercised runtime therefore pins torch==2.7.0, transformers==4.51.3, flash-attn==2.7.4.post1, and ninja==1.13.0, with no build isolation only for FlashAttention
  • RoboCasa 1.0.1 at b4684e6e and robosuite 1.5.2 at 5ce6643f are checked after installing the tested patch revisions

Validation

  • current-head H100 direct-wrapper parity passes for KettleBoiling (pretrain, seed 0) and ArrangeBreadBasket (pretrain, seed 3)
  • reset state, one-step state, all three RGB observations, five state observations, array metadata, and action order match exactly; every measured difference is 0
  • current-head success-protocol parity against the pinned GR00T MultiStepWrapper passes transient-mid-chunk, chunk-endpoint, and final-partial-chunk cases
  • legacy split=None and split="all" remain accepted, while rc365 remains restricted to pretrain and target
  • local validation passes Ruff, formatting, ty, and pytest (441 passed, 2 skipped)
  • all six GitHub checks pass

@MilkClouds

Copy link
Copy Markdown
Collaborator Author

@codex review

MilkClouds and others added 4 commits July 20, 2026 03:30
Ports the S2B integration onto the refactored benchmark: OSMesa-backed
VLA_EVAL_RENDER=cpu toggle (MuJoCo EGL needs a device platform Mesa
llvmpipe lacks; verified in the container with no GPU), deterministic
per-episode seeding, the hierarchical S2-plus-S1 policy server,
qualification-slice adapter parity (six tasks, recorded 32-step action
comparison, pass), and a Slurm qualification runner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MilkClouds

Copy link
Copy Markdown
Collaborator Author

Pushed the RC365 S2B integration onto this branch (a0ef7e3), ported to the refactored benchmark API:

  • CPU rendering: VLA_EVAL_RENDER=cpu routes MuJoCo through OSMesa (Mesa llvmpipe cannot serve MuJoCo's EGL device-platform requirement). Verified in the charliecloud-converted container with no GPU: all three RC365 cameras render. Motivation: CUDA compute and EGL readback on the same GPU abort nondeterministically in the NVIDIA driver (controlled A/B: 6/6 stable split vs 4/4 abort co-tenant), so CPU rendering restores one-GPU rollouts.
  • Hierarchical policy server (model_servers/rc365_s2b.py): the S2B System-2-plus-System-1 stack as one stateful policy; System 2 re-queried every 16-step chunk (gold sequence, global-only, random-valid, or MLLM planner).
  • Adapter parity extended from Kettle to the six qualification tasks (recorded 32-step action comparison, committed report, pass).
  • Qualification runner + Slurm template producing scorer-compatible dumps.

Full suite: 450 passed, 2 skipped. Follow-up work (GPU policy inference through the runner at scale) tracked on our side.

Pushed by Claude Fable 5 on behalf of @MilkClouds.

MilkClouds and others added 4 commits July 22, 2026 02:28
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drops the project-specific qualification runner, cluster launcher,
parity artifacts, and docker resource-mode changes from this PR; they
live in the consuming project. Keeps the render toggle, per-episode
seeding, the hierarchical policy server with its config and tests, the
CPU render check, and policy-initiated episode termination.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant