Skip to content

[noetic-devel] Refactoring tracker_with_cloud_node #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: noetic-devel
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 77 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,70 @@
# ultralytics_ros [![ROS-noetic Industrial CI](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-ci.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-ci.yml) [![ROS-noetic Docker Build Check](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-docker-build-check.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-docker-build-check.yml)
ROS package for real-time object detection and segmentation using the Ultralytics YOLO, enabling flexible integration with various robotics applications.
# ultralytics_ros
### Introduction
ROS/ROS 2 package for real-time object detection and segmentation using the Ultralytics YOLO, enabling flexible integration with various robotics applications.

| `tracker_node` | `tracker_with_cloud_node` |
| :------------: | :-----------------------: |
| <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/7ccefee5-1bf9-48de-97e0-a61000bba822" width="450px"> | <img src="https://github.com/Alpaca-zip/ultralytics_ros/assets/84959376/674f352f-5171-4fcf-beb5-394aa3dfe320" height="160px"> |

- The `tracker_node` provides real-time object detection and segmentation on incoming ROS image messages using the Ultralytics YOLO model.
- The `tracker_node` provides real-time object detection on incoming ROS/ROS 2 image messages using the Ultralytics YOLO model.
- The `tracker_with_cloud_node` provides functionality for 3D object detection by integrating 2D detections, mask image, LiDAR data, and camera information.

### Status
| ROS distro | Industrial CI | Docker |
| :--------: | :-----------: | :----: |
| ROS Melodic | [![ROS-melodic Industrial CI](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/melodic-ci.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/melodic-ci.yml) | [![ROS-melodic Docker Build Check](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/melodic-docker-build-check.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/melodic-docker-build-check.yml)
| ROS Noetic | [![ROS-noetic Industrial CI](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-ci.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-ci.yml) | [![ROS-noetic Docker Build Check](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-docker-build-check.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/noetic-docker-build-check.yml)
| ROS 2 Humble | [![ROS2-humble Industrial CI](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-ci.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-ci.yml) | [![ROS2-humble Docker Build Check](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-docker-build-check.yml/badge.svg)](https://github.com/Alpaca-zip/ultralytics_ros/actions/workflows/humble-docker-build-check.yml)

## Setup ⚙
### ROS Melodic
```bash
$ cd ~/{ROS_WORKSPACE}/src
$ GIT_LFS_SKIP_SMUDGE=1 git clone -b melodic-devel https://github.com/Alpaca-zip/ultralytics_ros.git
$ rosdep install -r -y -i --from-paths .
$ pip install pipenv
$ cd ultralytics_ros
$ pipenv install
$ pipenv shell
$ cd ~/{ROS_WORKSPACE} && catkin build
```
$ cd ~/catkin_ws/src
### ROS Noetic
```bash
$ cd ~/{ROS_WORKSPACE}/src
$ GIT_LFS_SKIP_SMUDGE=1 git clone -b noetic-devel https://github.com/Alpaca-zip/ultralytics_ros.git
$ rosdep install -r -y -i --from-paths .
$ python3 -m pip install -r ultralytics_ros/requirements.txt
$ cd ~/catkin_ws
$ cd ~/{ROS_WORKSPACE} && catkin build
```
### ROS 2 Humble
```bash
$ cd ~/{ROS2_WORKSPACE}/src
$ GIT_LFS_SKIP_SMUDGE=1 git clone -b humble-devel https://github.com/Alpaca-zip/ultralytics_ros.git
$ rosdep install -r -y -i --from-paths .
$ catkin build
$ python3 -m pip install -r ultralytics_ros/requirements.txt
$ cd ~/{ROS2_WORKSPACE} && $ colcon build
```
**NOTE**: If you want to download KITTI datasets, remove `GIT_LFS_SKIP_SMUDGE=1` from the command line.

## Run 🚀
### ROS Melodic & ROS Noetic
**`tracker_node`**
```
```bash
$ roslaunch ultralytics_ros tracker.launch debug:=true
```
**`tracker_node` & `tracker_with_cloud_node`**
```
```bash
$ roslaunch ultralytics_ros tracker_with_cloud.launch debug:=true
```
### ROS 2 Humble
**`tracker_node`**
```bash
$ ros2 launch ultralytics_ros tracker.launch.xml debug:=true
```
**`tracker_node` & `tracker_with_cloud_node`**
```bash
$ ros2 launch ultralytics_ros tracker_with_cloud.launch.xml debug:=true
```
**NOTE**: If the 3D bounding box is not displayed correctly, please consider using a lighter yolo model(`yolov8n.pt`) or increasing the `voxel_leaf_size`.

## `tracker_node`
Expand All @@ -48,16 +86,16 @@ For yolov8, you can choose `yolov8*.pt`, `yolov8*-seg.pt`.
- `tracker`: Tracking algorithms.
- `device`: Device to run the model on(e.g. cpu or cuda:0).
```xml
<arg name="device" default="cpu"/> <!-- cpu -->
<arg name="device" default="cpu"/>
```
```xml
<arg name="device" default="0"/> <!-- cuda:0 -->
<arg name="device" default="cuda:0"/>
```
- `classes`: List of class indices to consider.
```xml
<arg name="classes" default="[0, 1]"/> <!-- person, bicycle -->
<param name="classes" value="0, 1" value-sep=", "/> <!-- person, bicycle -->
```
See also: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/datasets/coco128.yaml
See also: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/coco128.yaml
- `result_conf`: Whether to plot the detection confidence score.
- `result_line_width`: Line width of the bounding boxes.
- `result_font_size`: Font size of the text.
Expand All @@ -66,15 +104,16 @@ For yolov8, you can choose `yolov8*.pt`, `yolov8*-seg.pt`.
- `result_boxes`: Whether to plot the bounding boxes.
### Topics
- Subscribed Topics:
- Image data from `input_topic` parameter. ([sensor_msgs/Image](https://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html))
- Image data from `input_topic` parameter. ([sensor_msgs/Image](https://github.com/ros2/common_interfaces/blob/humble/sensor_msgs/msg/Image.msg))
- Published Topics:
- Plotted images to `result_image_topic` parameter. ([sensor_msgs/Image](https://docs.ros.org/en/api/sensor_msgs/html/msg/Image.html))
- Plotted images to `result_image_topic` parameter. ([sensor_msgs/Image](https://github.com/ros2/common_interfaces/blob/humble/sensor_msgs/msg/Image.msg))
- Detected objects(2D bounding box, mask image) to `result_topic` parameter. (ultralytics_ros/YoloResult)
```
std_msgs/Header header
vision_msgs/Detection2DArray detections
sensor_msgs/Image[] masks
```

## `tracker_with_cloud_node`
### Params
- `camera_info_topic`: Topic name for camera info.
Expand All @@ -99,16 +138,40 @@ For yolov8, you can choose `yolov8*.pt`, `yolov8*-seg.pt`.
- Detected cloud points to `/detection_cloud` topic. ([sensor_msgs/PointCloud2](https://docs.ros.org/en/api/sensor_msgs/html/msg/PointCloud2.html))
- Detected objects(3D bounding box) to `yolo_3d_result_topic` parameter. ([vision_msgs/Detection3DArray](http://docs.ros.org/en/lunar/api/vision_msgs/html/msg/Detection3DArray.html))
- Visualization markers to `/detection_marker` topic. ([visualization_msgs/MarkerArray](https://docs.ros.org/en/api/visualization_msgs/html/msg/MarkerArray.html))

## Docker with KITTI datasets 🐳
[![dockeri.co](https://dockerico.blankenship.io/image/alpacazip/ultralytics_ros)](https://hub.docker.com/r/alpacazip/ultralytics_ros)

### Docker Pull & Run
**ROS Melodic**
```bash
$ docker pull alpacazip/ultralytics_ros:melodic
$ docker run -p 6080:80 --shm-size=512m alpacazip/ultralytics_ros:melodic
```
**ROS Noetic**
```bash
$ docker pull alpacazip/ultralytics_ros:noetic
$ docker run -p 6080:80 --shm-size=512m alpacazip/ultralytics_ros:noetic
```
**ROS 2 Humble**
```bash
$ docker pull alpacazip/ultralytics_ros:humble
$ docker run -p 6080:80 --shm-size=512m alpacazip/ultralytics_ros:humble
```
### Run tracker_node & tracker_with_cloud_node
**ROS Melodic**
```bash
$ roscd ultralytics_ros && pipenv shell
$ roslaunch ultralytics_ros kitti_predict_with_cloud.launch
$ cd ~/catkin_ws/src/ultralytics_ros/rosbag && rosbag play kitti_2011_09_26_drive_0106_synced.bag --clock --loop
```
**ROS Noetic**
```bash
$ roslaunch ultralytics_ros kitti_tracker_with_cloud.launch
$ cd ~/catkin_ws/src/ultralytics_ros/rosbag && rosbag play kitti_2011_09_26_drive_0106_synced.bag --clock --loop
```
**ROS 2 Humble**
```bash
$ ros2 launch ultralytics_ros kitti_tracker_with_cloud.launch.xml
$ cd ~/colcon_ws/src/ultralytics_ros/ros2bag && ros2 bag play kitti_2011_09_26_drive_0106_synced --clock --loop
```