README.md

RoboLab Documentation

How RoboLab Works

RoboLab dynamically combines tasks with user-specified robot, observations, actions, and simulation parameters at environment registration time. The core concepts are:

• Objects — USD object assets with physics properties for manipulation
• Scenes — USD-based environments containing objects, fixtures, and spatial layout
• Tasks — Language instructions, termination criteria, and scene bindings
• Subtask Checking — Granular progress tracking within tasks
• Conditionals — Predicate logic for defining success/failure conditions
• Event Tracking — Monitoring task-relevant events during execution
• Task Libraries — Managing task collections, generating metadata, and viewing statistics
• Robots — Robot articulation configs, actuators, and action spaces
• Cameras — Scene cameras and robot-attached cameras
• Lighting — Scene lighting (sphere, directional, and custom lights)
• Backgrounds — HDR/EXR dome light backgrounds
• Environment Registration — How tasks are combined with robot/observation/action configs into runnable Gymnasium environments
• Environment Generation — Contact sensor creation, subtask trackers, and runtime environment internals
• Inference Clients — Built-in policy clients and server setup instructions (OpenPI, GR00T)
• Running Environments — Creating environments, evaluation scripts, CLI reference, and robustness testing
• Data Storage and Output — Output directory structure, HDF5 layout, and episode result fields
• Analysis and Results Parsing — Scripts for summarizing, comparing, and auditing experiment results

Development Workflow

If you're building a completely new benchmark and workflow, follow the steps below in order. Otherwise, pick whichever applies to your use case.

Creating new assets, tasks, and benchmarks

1. Creating New Objects — Author USD object assets with rigid body, collision, and friction properties. Includes pipeline for catalog generation, screenshots, and physics tuning.
2. Creating New Scenes — Compose objects into USD scenes using IsaacSim. Includes settling, metadata generation, and screenshot utilities.
3. Creating New Tasks — Define task dataclasses with language instructions, termination criteria, and scene bindings. Tasks can live in your own repository.
4. Managing Task Libraries — Organize tasks into collections, generate metadata (JSON, CSV, README), and compute statistics.

Configuring robots, cameras, lighting, and backgrounds

• Robots — Define or customize robot articulation, actuators, and action spaces. Use built-in configs (DROID, Franka) or bring your own from IsaacLab.
• Cameras — Set up scene cameras and robot-attached cameras (e.g., wrist cameras).
• Lighting — Configure scene lighting for evaluation or robustness testing.
• Backgrounds — Set HDR/EXR dome light backgrounds for realistic scene rendering.

Evaluating a new policy against the benchmark

1. Setting Up Environment Registration — Register tasks with your robot/observation/action/simulation settings. For DROID with joint-position actions, the built-in registration can be used directly.
2. Evaluating a New Policy — Implement an inference client for your model and run multi-task evaluation. Everything can live in your own separate repository.

AI Workflows

• Scene Generation — Generate USD scenes from natural language using the /robolab-scenegen Claude Code skill. See skills/robolab-scenegen/.
• Task Generation — Generate task files from natural language using the /robolab-taskgen Claude Code skill. See skills/robolab-taskgen/.

Running and debugging

• Running Environments — Creating environments, evaluation scripts, CLI reference, and robustness testing
• Data Storage and Output — Output directory structure, HDF5 layout, and episode result fields
• Analysis and Results Parsing — Scripts for summarizing, comparing, and auditing experiment results
• Debugging — Verbose/debug flags, world state inspection, and diagnostic scripts