Viewpoint-Agnostic Grasping Pipeline

Language-guided, occlusion-robust grasping for Boston Dynamics Spot — deployed on real robot

Key Features

• End-to-End Pipeline: Natural language → segmentation → 3D completion → grasp → execution
• Isaac Sim Simulation: Spot robot with 6 RGB-D cameras and ROS2 bridge
• NVBlox Integration: GPU-accelerated TSDF mapping for real-time 3D reconstruction
• Semantic Segmentation: GroundingDINO + SAM2 (video tracker) + RAM
• Two-Stage Point Cloud Completion: MGPC (multimodal) + PoinTr (patch-wise densification)
• 6-DOF Grasp Generation: PyGPG with execution-aware heuristic selection and collision checking
• Real Robot Deployment: Spot arm with 90% success rate on cluttered tabletop (IROS 2026)
• Modular Architecture: Skill-based interfaces for all AI models

Requirements

• Ubuntu 22.04.5 LTS
• CUDA 12.8+ (Recommended: RTX 3000+)
• ~100 GB of free disk

Installation

0. Clone the repo

git clone --recurse-submodules https://github.com/EESC-LabRoM/isaac_dl_grasp.git
cd isaac_dl_grasp

1. Install each environment

This project is divided between simulation environment and ROS environment, one independent from another. The following sections describre the intalation for each one.

Also, this project hevely relly in just recipes. Install it with:

# just instalation
sudo snap install just --classic

1.1. Simulation environment (host machine)

Run the install script to set up the simulation via uv:

# Isaac Sim + Lab instalation
just install-sim

Important

Isaac Sim already has internal ROS2 libraries. Do not source your local ROS2 installation (source /opt/ros/<distro>/setup.bash) in .bashrc — this causes conflicts when running Isaac Sim.

1.2. Isaac ROS Container (ML nodes + NVBlox)

The ROS interface works via Isaac ROS container. Follow the instructions on Isaac ROS CLI Installation Guide to install it. Notice that the initialization must be made with Docker.

Then set up the environment inside the container. The just recipes will handle the ISAAC_ROS_WS variable:

# Copy dockerfiles to Isaac ROS CLI folder
just ros-cp-dockerfiles

# Start the Isaac ROS container (this may take a while in the first run, around 1 hour)
just ros-ws

On another terminal, after the initialization:

# Creates .venv/ros_env with all dependencies via uv
just ros-install-env

# Download AI models
just ros-download-models

# Build the ROS2 workspace
just ros-build-ws

Usage

The main scripts and workflow can run with just recipes. Run just while in the project root to see all availale commands:

# This will show all the shortcuts in the justfile
just

Main Wokflow

Viewpoint-Agnostic main workflow on simulated environment.

#=========================# TERMINAL 1 #=========================#
#                           Simulation

just run-spot-sim

#=========================# TERMINAL 2 #=========================#
#                          ROS pipeline

just ros-full-pipeline seg_cam:=frontleft

#=========================# TERMINAL 3 #=========================#
#                            Controll

just ros-find "power drill"         # 1. set object target
just ros-seg-now                    # 2. force segmentation frame
just ros-grasp-predict              # 3. run MGPC + PoinTr + GPG
just ros-move-to                    # 4. walk robot to face target
just ros-pick                       # 5. approach → grasp → retrieve

Project Structure

isaac_dl_grasp/
├── spot_sim/                       # Simulation environment (Isaac Sim/Lab)
│   ├── assets/                       # Robot + object assets (Spot, drill, etc.)
│   ├── external/                     # Git submodules (host-side)
│   │   ├── relic/                      # Spot URDF/USD assets
│   │   └── curobo/                     # Motion planning & IK
│   ├── scripts/
│   │   ├── spot_isaacsim/            # Current main: Isaac Sim + ROS2 bridge
│   │   │   ├── play.py                 # Entry point (just run-spot-sim)
│   │   │   ├── scene/                  # Scene setup (builder.py, cameras.py)
│   │   │   ├── omnigraph/              # ROS2 bridge (builder.py — OmniGraph)
│   │   │   ├── arm_controller.py       # Lula IK arm controller
│   │   │   ├── grasp_executor.py       # Grasp execution state machine
│   │   │   └── spot_config/            # Robot configuration (YAML)
│   │   └── tools/                    # Utility scripts (install, convert, patch)
│   └── pyproject.toml                # uv dependency management (simulation)
├── IsaacROS/                       # Isaac ROS workspace (ML + NVBlox)
│   ├── external/                     # AI model submodules
│   │   ├── Grounded-SAM-2/             # SAM2 + GroundingDINO
│   │   ├── PoinTr/                     # Point cloud completion
│   │   ├── MGPC/                       # Multi-view point cloud
│   │   ├── recognize-anything/         # RAM (tag generation)
│   │   └── pygpg/                      # Grasp pose generation (C++)
│   ├── modules/                      # Modular architecture
│   │   ├── models_interface/           # Skill-based AI model interfaces
│   │   └── grasp_module/              # Grasp pose generation & selection
│   ├── ros_ws/                       # ROS2 workspace
│   │   └── src/
│   │       ├── spot_nvblox/            # NVBlox launch files
│   │       └── agn_grasp/             # ML inference ROS2 nodes
│   ├── scripts/tools/                # Container utility scripts
│   └── pyproject.toml                # uv dependency management (container)
└── justfile                        # Task runner (run 'just' for all commands)

Configuration & Development Notes

• Environment: Use just run-spot-sim to run the simulation on the Python environment (via uv run).
• Isaac ROS: NVBlox and ML nodes run in a Docker container (isaac_ros_dev_container). Use just ros-ws to activate.
• ML Dependencies: Managed via uv and IsaacROS/pyproject.toml. Run just ros-install-env inside the container for one-time setup.
• Troubleshooting:
  • RMW mismatch (topics not visible): Both host and container must use export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp with the same ROS_DOMAIN_ID. Run ros2 daemon stop && ros2 daemon start in the container.
  • Isaac Sim CUDA warnings: Safe to ignore.
  • ROS2 conflicts: Do NOT source local ROS2 installation in .bashrc — Isaac Sim bundles its own ROS2 libraries.

License

MIT License