dl_grasping_loco_manipulation_2025

Deep learning pipeline for grasping with the Boston Dynamics Spot robot. Developed at the Mobile Robotics Group, EESC (São Carlos Engineering School).

The project covers simulation-based dataset generation, neural network training for grasp selection, and real robot deployment using Spot's arm and gripper camera.

• Genesis docs: genesis-world.readthedocs.io
• Genesis repo: Genesis GitHub

Repository Structure

├── urdf/             # Robot description files (URDF, meshes)
├── scripts/          # Genesis simulation scripts
│   └── spot_gripper/ # Gripper grasping experiments, including normal-aligned grasping and D2NT validation
├── dataset/          # Dataset extraction notebooks and 3D bottle models
├── grasp_selection/  # Grasp selection neural network training
└── spot_deploy/      # Real robot deployment pipeline (YOLO + D2NT + GraspNN)
    ├── evaluation/   # Full integration pipeline
    ├── images/       # Camera data from Spot's gripper
    └── tuning/       # Parameter tuning scripts

Setup

Simulations run on Genesis. See the installation guide.

Using conda:

conda create -n genesis_env python=3.12
conda activate genesis_env
pip install genesis-world

Using pyenv:

pyenv virtualenv 3.12 genesis_env
pyenv activate genesis_env
pip install genesis-world

An NVIDIA GPU is recommended. Install CUDA drivers for better performance.

Real Robot

Deployment uses the Boston Dynamics Spot SDK:

• Python examples
• Protobuf API

Normal maps on the real robot are generated using D2NT. In simulation, Genesis provides native tools for this.

Genesis Documentation

• User Guide
• API Reference

Contributing

Open an issue or pull request if you have suggestions or improvements.