CRAIP: Creating Robot Artificial Intelligence Project | Team 5

This repository contains Team 5's submission for Seoul National University's "How to Make a Robot with Artificial Intelligence (M2177.002600), Fall 2025" course.

🎯 Project Overview

CRAIP is developed using the Language-Image-based Mobile Robot (LIMBO) framework, designed to enable robots to perform autonomous navigation tasks based on language instructions. Our project implements a complete robotic system integrating perception, planning, and control modules for the Unitree Go1 quadruped robot in a Gazebo simulation environment.

Key Features:

Natural language command processing for robot navigation
Vision-based object detection and perception
Path planning and tracking control
Autonomous gait control for quadruped locomotion
ROS2-based modular architecture

🔧 System Architecture

Our system consists of several integrated ROS2 packages working together to enable autonomous navigation:

Core Components

Perception Pipeline: Real-time object detection and scene understanding
Language Understanding: Processing natural language commands
Path Planning: A* algorithm-based path planning with collision avoidance
Motion Control: MPPI-based path tracking and gait controller
Simulation Environment: Gazebo-based virtual testing environment

⚙️ Prerequisites

Operating System: Ubuntu 24.04
ROS2 Distribution: Jazzy Jalisco
Gazebo: Harmonic

📥 Setup & Installation

Step 1: Clone the Repository

cd ~
git clone https://github.com/yanghyeonseo/make_ai_robot_team5.git
cd make_ai_robot_team5

Step 2: Install Dependencies

Use rosdep to automatically install all required dependencies:

sudo apt update && sudo apt upgrade

sudo apt-get install python3-rosdep
sudo rosdep init
rosdep update

cd ~/make_ai_robot_team5
rosdep install -i --from-path src --rosdistro $ROS_DISTRO -y

This command installs:

Gazebo Harmonic and ROS2 integration (ros_gz_sim, ros_gz_bridge)
Core ROS2 packages (rclcpp, rclpy, tf2, ros2_control, etc.)
Required system libraries (Eigen, Boost, OpenCV, etc.)

Step 3: Build the Workspace

colcon build

The build should complete successfully with all packages compiled:

Summary: 12 packages finished [XXs]

Step 4: Source the Workspace

source install/setup.bash

For automatic sourcing on every terminal session, add to ~/.bashrc:

echo "source ~/make_ai_robot/install/setup.bash" >> ~/.bashrc
source ~/.bashrc

Workspace Structure

After successful build:

make_ai_robot/
├── src/
│   ├── astar_planner/
│   ├── custom_interfaces/     
│   ├── environment/       
│   ├── go1_simulation/
│   ├── language_command_handler/
│   ├── localization/
│   ├── module_test/
│   ├── path_tracker/
│   ├── perception/
│   ├── ros2_unitree_legged_controller/
│   ├── ros2_unitree_legged_msgs/
│   └── unitree_guide2/
├── build/                  # Temporary build files (auto-generated)
├── install/                # Compiled packages (auto-generated)
├── log/                    # Build logs (auto-generated)
└── docker/                 # Docker configuration

🚀 Running the System

System Overview

The system runs across 7 terminals, each with a specific role in the autonomous navigation pipeline.

Terminal Setup & Execution

Terminal 1: Gazebo Simulation

Launches the Go1 robot in Gazebo simulation environment.

ros2 launch go1_simulation go1.gazebo.launch.py

Role: Physics simulation, sensor simulation (LiDAR, camera, IMU)

Terminal 2: Robot Controller

Runs the high-level locomotion FSM controller. After launching, press keys 1→2→5 in sequence:

Key 1: Initialize robot
Key 2: Stand up
Key 5: Motion control mode

ros2 run unitree_guide2 junior_ctrl

Role: Robot gait control and motion state management

Terminal 3: Localization Module

Estimates robot pose from sensor data (LiDAR, IMU) using particle filter.

ros2 launch localization localizer.launch.py

Role: Robot self-localization, publishes /go1_pose topic

Terminal 4: Path Planner

Computes collision-free path from current position to goal using A* algorithm.

ros2 launch astar_planner astar_planner.launch.py use_gazebo:=true

Role: Path planning, publishes /local_path topic

Terminal 5: Path Tracker

Follows the planned path using MPPI control algorithm, generates velocity commands.

ros2 launch path_tracker path_tracker_launch.py

Role: Trajectory tracking, publishes /cmd_vel commands to move robot

Terminal 6: Interface Viewer

Visualizes detection results and system status in real-time interface.

ros2 launch module_test interface_viewer.launch.py

Role: Perception visualization and monitoring

Terminal 7: Language Command Handler

Processes natural language commands and converts them to robot actions. Requires Anaconda environment.

Setup (one-time):

# Download and install Anaconda (skip if already installed)
cd ~/Downloads
wget https://repo.anaconda.com/archive/Anaconda3-2025.06-1-Linux-x86_64.sh
bash Anaconda3-2025.06-1-Linux-x86_64.sh

# Edit ~/.bashrc and add OpenAI API key
export OPENAI_API_KEY='sk-your-api-key-here'

# Create conda environment
conda create --name language_command_handler python=3.12.3 -y
conda activate language_command_handler
cd ~/make_ai_robot/src/language_command_handler
pip install -r requirements.txt

Execution:

conda activate language_command_handler
ros2 launch language_command_handler start_command_handler.launch.py

Role: Language understanding and command execution

Sending Commands to the Robot

Once all 7 terminals are running, send commands via ROS2 service:

# Example: Move forward
ros2 service call /language_command custom_interfaces/srv/LanguageCommand "{command: 'Go forward.'}"

# Other examples
ros2 service call /language_command custom_interfaces/srv/LanguageCommand "{command: 'Move the box to the red zone.'}"
ros2 service call /language_command custom_interfaces/srv/LanguageCommand "{command: 'Stop in front of the red cone.'}"
ros2 service call /language_command custom_interfaces/srv/LanguageCommand "{command: 'Go to the restroom.'}"

Note: Commands are processed by the LLM and mapped to executable ROS2 actions based on language_command_handler/config/command_handler_config.yaml.

🛠️ Module Design: 30 Points

Build custom implementations of two core navigation modules:

Module 1: Localization (20 pts)

Package: localization

Goal: Estimate robot pose from sensor data using particle filter.

Input Topics:

/scan - LiDAR 2D scan data
/imu_plugin/out - IMU measurements
/map - Occupancy grid map

Output Topics:

/go1_pose - Robot pose (geometry_msgs/msg/PoseStamped)
/tf - Transform frames (map → odom → base)

Algorithms: Particle Filter, Scan Matching (ICP), Sensor Fusion

Results: Successfully implemented with exceptional accuracy. Achieved ATE (Absolute Trajectory Error) of 0.03m, demonstrating highly accurate localization performance across all test scenarios.

Evaluation: Tested on 3 ROS bags with Absolute Trajectory Error (ATE) metric. ✅ Full 20 points awarded.

Module 2: Perception (10 pts)

Package: perception

Goal: Real-time object detection and distance estimation.

Input Topics:

/camera_top/image - RGB camera feed
/camera_top/depth - Depth image
/camera_top/points - Point cloud

Output Topics:

/camera/detections/image - Annotated image with bounding boxes
/detections/labels - Object class (string)
/detections/distance - Distance to object (float32)
/robot_dog/speech - "bark" when edible object is centered, else "None"

Results: Successfully implemented robust object detection with accurate distance estimation. System correctly detects objects and generates appropriate bark responses.

Evaluation: Tested on 2 ROS bags with screen recordings showing detection and bark timing validation. ✅ Full 10 points awarded.

⚔️ Competition: 6 Missions (70 points)

Overview: 30 minutes to complete 6 missions using only natural language commands. Total score: 70 points (6 missions × 10 points + 10 bonus points).

Mission 1: Navigate to Toilet (10 pts)

Navigate to the toilet location. Robot must bark when it faces the toilet directly.

Strategy: Pre-explore map, hard-code toilet position
Approach: Path planning → Tracking
Reference Images:

Mission 2: Find Edible Food (10 pts)

Search environment and bark in front of edible food (apple/banana/pizza).

Challenge: Food positions vary between competitions. YOLOv8 model needs fine-tuning on custom food datasets to improve detection accuracy.
Strategy: Active search using cameras for edible items. Fine-tune YOLOv8 on collected food images to ensure robust detection across different lighting and angles.
Approach: Exploration → Detection → Verification
Reference Images:

Mission 3: Go to Colored Cone (10 pts)

Navigate to a specified colored cone (red/green/blue) and bark.

Challenge: Cone colors and order may change
Strategy: Train color detection, navigate using perception
Approach: Detection → Path planning → Alignment
Reference Images:

Mission 4: Push Box to Goal (10 pts)

Push delivery box to red goal area using robot body.

Challenge: Box starting position changes between runs
Strategy: Detect box position, plan pushing path
Approach: Box detection → Goal planning → Push execution
Reference Images:

Mission 5: Enter Empty Room (10 pts)

Move to an empty room that doesn't have a stop sign.

Challenge: Stop sign position changes
Strategy: Detect stop sign presence, choose correct room
Approach: Sign detection → Room selection → Navigation
Reference Images:

Mission 6: Rotate Around the Nurse (10 pts)

Find the nurse in the break room and rotate around her.

Challenge: Nurse position varies slightly between competitions
Strategy: Detect human presence, establish circular path around detected person
Approach: Human detection → Path planning → Circular navigation execution
Reference Images:

📚 Resources & References

Baseline Code

make_ai_robot: Baseline code repository - GitHub Repository

Key Technologies

ROS2: Robotic Operating System - Official Documentation
Gazebo Harmonic: Physics simulation - Documentation
Unitree Go1: Quadruped robot - Official Repository

Machine Learning & Vision

YOLOv8: Object detection - Repository
OpenAI API: Language model integration - Documentation

Algorithms

A* Algorithm: Path planning
MPPI (Model Predictive Path Integral): Trajectory tracking
Particle Filter: Monte Carlo Localization (MCL)
Scan Matching: Iterative Closest Point (ICP)

🏆 Competition Results

Final Results: Team 5 successfully completed 5 out of 6 missions (all missions except Mission 2) in just 8 minutes and 30 seconds.

Overall Ranking: 2nd Place out of 10 teams

Points Achieved: 50 points (Mission 1, 3, 4, 5, 6 completed) + 30 points (Module Design) = 80 points total

Mission Live Stream

Watch the live competition video:

👨‍💻 Team Members

Team 5: Sanghyun Park, Hyeonseo Yang, Jinseok Choi

Name	Role	Contributions
Sanghyun Park	Perception Lead	YOLOv8 object detection, distance estimation, bark response logic
Hyeonseo Yang	Localization Lead	Particle filter implementation, sensor fusion, ATE optimization
Jinseok Choi	Infrastructure & DevOps	ROS2 setup, package coordination, module integration

📬 Contact

Sanghyun Park: qkrtkdgus39@snu.ac.kr
Hyeonseo Yang: yanghs1018@snu.ac.kr
Jinseok Choi: gates02@snu.ac.kr

For questions about the project, course, or technical issues, please contact the team members above.

Name		Name	Last commit message	Last commit date
Latest commit History 102 Commits
docker		docker
images		images
models		models
src		src
.gitignore		.gitignore
README.md		README.md

euntimes2/make_ai_robot_team5

Folders and files

Latest commit

History

Repository files navigation