Brian/deep object detection (#26)

* Initial implementation of deep object detection

* switched to onnx inferencing

* Added onnxruntime gpu to cmake

* pre-commit fixes

* Added ort_backend integration, Fixed detection parsing issues, fixed config issues and removed other launch files

TODO: Allow dynamic batch sizes. Optimize!

* Initial deep_ort_gpu plugin with internal library issues

* Functioning onnxruntime with CUDA as EP

* Initial deep_ort_gpu plugin with internal library issues

* Functioning onnxruntime with CUDA as execution provider

* Removed requirement for cuda toolkit

* batch size and other stuff?

* hopefully fixed segfault

* working cuda inference

* working tensorrt ep

* new stuff

* cleaning up cmake

* fixed cuda

* added libnvinfer, trt ep should work

* caching

* should be working

* fixed batch size

* optional engine caching

* some cleanup

* tensorrt version

* precommit and cleanup

* added lifecycle node

* loading backend at runtime

* working detection

* dynamic tensor output and configurable activation

* addressing pr comments

* node improvements

* some changes

* docstrings and remapping

* Fixed yolov8 layout detection bug

* Added ImageMarker publisher for foxglove

* added tests

* precommit

* opencv build fix

* start the deep clean, get rid of batching

* get rid of main, ort backend io bind

* sorted out the params situation

* sorted out the param situation

* nuke backend manager, clean up params

* deep node base inheritance, multi raw image support, cleanup

* fixed msg naming

* removed multi output, clean up readme

* update readme

* fix tests

* why was this here

* fix logging?

* last changes, and i switched the yolo dimensions xd

* fix build

* pls fix build error

* test

* cv bridge?

---------

Co-authored-by: lucasreljic <lucas.reljic@gmail.com>
Co-authored-by: Eddy Zhou <edsteredward@gmail.com>

Author: 3 people

.devcontainer/Dockerfile

@@ -34,7 +34,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     && rm -rf /var/lib/apt/lists/*
 ENV DEBIAN_FRONTEND=interactive
 
-# Install Tensorrt Runtime
 RUN curl -fsSL -o cuda-keyring_1.1-1_all.deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb \
     && dpkg -i cuda-keyring_1.1-1_all.deb \
     && apt-get update && apt-get install -y --no-install-recommends \
@@ -59,9 +58,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     curl \
     && rm -rf /var/lib/apt/lists/*
 
-# Install pre-commit
-RUN python3 -m pip install --no-cache-dir pre-commit==3.8.0
-
 # Set working dir (matches VSCode workspace)
 WORKDIR /deep_ros_ws
 
@@ -117,6 +113,9 @@ RUN groupadd --gid ${USER_GID} ${USERNAME} \
 # Set the default user. Omit if you want to keep the default as root.
 USER $USERNAME
 
+# Install pre-commit for the user
+RUN python3 -m pip install --no-cache-dir --user pre-commit==3.8.0
+
 # Install Claude Code natively
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 RUN curl -fsSL https://claude.ai/install.sh | bash

DEVELOPING.md

@@ -26,6 +26,75 @@ This project includes VS Code dev container configurations for easy ROS2 develop
 - **Build tools**: Includes `colcon` and `rosdep` for ROS development
 - **Extensions**: C++, CMake, Python, and XML support pre-installed
 
+### Stopping Containers
+
+After using "Rebuild and Reopen in Container", you can stop containers using:
+
+**Method 1: VS Code Command (Recommended)**
+- Press `Ctrl+Shift+P` (or `Cmd+Shift+P` on Mac)
+- Type: `Dev Containers: Reopen Folder Locally`
+- This closes the container and returns to local mode
+
+**Method 2: Stop Container**
+- Press `Ctrl+Shift+P`
+- Type: `Dev Containers: Stop Container`
+- This stops the container but keeps it available for later use
+
+**Method 3: Using Docker Commands**
+From a terminal (outside the container):
+
+```bash
+# List running containers
+docker ps
+
+# Stop a specific container by name
+docker stop <container_name>
+
+# Stop all dev containers
+docker ps -q --filter "name=vsc-deep_ros" | xargs -r docker stop
+
+# Stop and remove containers
+docker stop <container_name>
+docker rm <container_name>
+```
+
+**Method 4: Close VS Code Window**
+- Simply closing the VS Code window will stop the container when you exit
+- The container will remain stopped until you reopen the folder in container mode
+
+### Restarting Containers
+
+After stopping a container, you can restart it using:
+
+**Method 1: VS Code Command (Recommended)**
+- Press `Ctrl+Shift+P` (or `Cmd+Shift+P` on Mac)
+- Type: `Dev Containers: Reopen in Container`
+- This will start the existing container or create a new one if needed
+
+**Method 2: Rebuild and Reopen**
+- Press `Ctrl+Shift+P`
+- Type: `Dev Containers: Rebuild and Reopen in Container`
+- Use this if you want to rebuild the container from scratch (e.g., after Dockerfile changes)
+
+**Method 3: Using Docker Commands**
+From a terminal (outside the container):
+
+```bash
+# List all containers (including stopped)
+docker ps -a
+
+# Start a stopped container by name
+docker start <container_name>
+
+# Start and attach to container
+docker start <container_name>
+docker attach <container_name>
+```
+
+**Method 4: Reopen Folder in VS Code**
+- If VS Code detects a devcontainer configuration, it will prompt you to "Reopen in Container"
+- Click the notification or use the command palette option
+
 ### Common Commands
 
 Inside the container, you can do ros2 commands, colcon commands, rosdep, etc.

deep_msgs/CMakeLists.txt

@@ -27,7 +27,7 @@ find_package(rosidl_default_generators REQUIRED)
 
 rosidl_generate_interfaces(${PROJECT_NAME}
   "msg/MultiImage.msg"
-  "msg/MultiImageRaw.msg"
+  "msg/MultiImageCompressed.msg"
   DEPENDENCIES std_msgs sensor_msgs
 )
 

deep_msgs/msg/MultiImage.msg

@@ -1,5 +1,3 @@
 # MultiImage.msg
-# A message that carries multiple compressed images together
-
 std_msgs/Header header
-sensor_msgs/CompressedImage[] images
+sensor_msgs/Image[] images

deep_msgs/msg/MultiImageCompressed.msg

@@ -0,0 +1,5 @@
+# MultiImageCompressed.msg
+# A message that carries multiple compressed images together
+
+std_msgs/Header header
+sensor_msgs/CompressedImage[] images

deep_msgs/msg/MultiImageRaw.msg

@@ -0,0 +1,112 @@
+# Copyright (c) 2025-present WATonomous. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cmake_minimum_required(VERSION 3.22)
+project(deep_object_detection)
+
+if(NOT CMAKE_CXX_STANDARD)
+  set(CMAKE_CXX_STANDARD 17)
+endif()
+
+if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wall -Wextra -Wpedantic)
+endif()
+
+find_package(ament_cmake REQUIRED)
+find_package(rclcpp REQUIRED)
+find_package(rclcpp_components REQUIRED)
+find_package(rclcpp_lifecycle REQUIRED)
+find_package(sensor_msgs REQUIRED)
+find_package(cv_bridge REQUIRED)
+find_package(vision_msgs REQUIRED)
+find_package(visualization_msgs REQUIRED)
+find_package(deep_core REQUIRED)
+find_package(deep_msgs REQUIRED)
+find_package(pluginlib REQUIRED)
+find_package(rcl_interfaces REQUIRED)
+find_package(OpenCV REQUIRED COMPONENTS core imgproc)
+
+add_library(deep_object_detection_lib SHARED
+  src/generic_postprocessor.cpp
+  src/image_preprocessor.cpp
+  src/deep_object_detection_node.cpp
+)
+
+target_include_directories(deep_object_detection_lib PUBLIC
+  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+  $<INSTALL_INTERFACE:include>
+)
+
+target_link_libraries(deep_object_detection_lib
+  PUBLIC
+    ${rclcpp_TARGETS}
+    ${rclcpp_lifecycle_TARGETS}
+    ${deep_core_TARGETS}
+    ${pluginlib_TARGETS}
+    ${vision_msgs_TARGETS}
+    ${deep_msgs_TARGETS}
+    ${visualization_msgs_TARGETS}
+    deep_core::deep_core_lib
+  PRIVATE
+    ${rclcpp_components_TARGETS}
+    ${sensor_msgs_TARGETS}
+    ${cv_bridge_TARGETS}
+    ${rcl_interfaces_TARGETS}
+    ${OpenCV_LIBRARIES}
+)
+
+target_include_directories(deep_object_detection_lib SYSTEM PUBLIC ${OpenCV_INCLUDE_DIRS})
+
+rclcpp_components_register_nodes(deep_object_detection_lib "deep_object_detection::DeepObjectDetectionNode")
+
+install(TARGETS deep_object_detection_lib
+  ARCHIVE DESTINATION lib
+  LIBRARY DESTINATION lib
+  RUNTIME DESTINATION bin
+)
+
+install(DIRECTORY include/
+  DESTINATION include
+)
+
+install(DIRECTORY config launch
+  DESTINATION share/${PROJECT_NAME}
+)
+
+if(BUILD_TESTING)
+  find_package(deep_test REQUIRED)
+
+  # Unit tests
+  add_deep_test(test_deep_object_detection_node test/test_deep_object_detection_node.cpp
+    LIBRARIES
+      deep_object_detection_lib
+      deep_core::deep_core_lib
+  )
+  # Set explicit timeout for unit test
+  set_tests_properties(test_deep_object_detection_node PROPERTIES TIMEOUT 10)
+
+  # Launch tests disabled by default
+  # Set ENABLE_LAUNCH_TESTS=1 to enable them
+  if(DEFINED ENV{ENABLE_LAUNCH_TESTS} AND "$ENV{ENABLE_LAUNCH_TESTS}" STREQUAL "1")
+    find_package(launch_testing_ament_cmake REQUIRED)
+    message(STATUS "Launch tests enabled (CPU/GPU/TensorRT)")
+    add_deep_launch_test(test/launch_tests/test_deep_object_detection_cpu_backend.py TIMEOUT 60)
+    add_deep_launch_test(test/launch_tests/test_deep_object_detection_gpu_backend.py TIMEOUT 60)
+    add_deep_launch_test(test/launch_tests/test_deep_object_detection_tensorrt_backend.py TIMEOUT 60)
+  else()
+    message(STATUS "Launch tests disabled by default (set ENABLE_LAUNCH_TESTS=1 to enable)")
+  endif()
+endif()
+
+ament_package()

deep_object_detection/CMakeLists.txt

@@ -0,0 +1,127 @@
+# Deep Object Detection Node
+
+ROS 2 node for deep learning object detection using ONNX-compatible models.
+
+This node provides model-agnostic object detection for ROS 2, supporting any ONNX-compatible detection model with configurable preprocessing, postprocessing, and multi-camera batch processing. All processing is lifecycle-managed for clean startup/shutdown.
+
+## Architecture
+
+```
+┌──────────────────────────────────────┐        ┌──────────────────────────────────────┐
+│ MultiImage / MultiImageCompressed    │ ─────► │ DeepObjectDetectionNode              │
+│ (multi-camera)                       │        │ (lifecycle node)                     │
+└──────────────────────────────────────┘        │  • Decode & preprocess               │
+                                                │  • Batch inference                   │
+                                                │  • Postprocess & NMS                 │
+                                                └───────────────┬──────────────────────┘
+                                                                │ publishes
+                                                                ▼
+┌──────────────────────────────────────┐        ┌──────────────────────────────────────┐
+│ Backend Plugins                      │ ◄───── │ Detection2DArray                     │
+│  • onnxruntime_cpu                   │  uses  │ ImageMarker (optional)               │
+│  • onnxruntime_gpu (CUDA/TRT)        │        └──────────────────────────────────────┘
+└───────────────┬──────────────────────┘
+                │ loads
+                ▼
+┌──────────────────────────────────────┐
+│ ONNX Model + YAML Config             │
+└──────────────────────────────────────┘
+
+
+```
+
+## Running
+
+```bash
+ros2 launch deep_object_detection deep_object_detection.launch.yaml \
+  config_file:=/path/to/config.yaml
+```
+
+Or run directly:
+
+```bash
+ros2 run deep_object_detection deep_object_detection_node \
+  --ros-args \
+  --params-file /path/to/config.yaml
+```
+
+## Parameters
+
+### Required
+
+- **`model_path`** (string): Absolute path to ONNX model file (e.g., `/workspaces/deep_ros/yolov8m.onnx`)
+- **`input_topic`** (string): MultiImage/MultiImageCompressed topic name to subscribe to
+
+### Model Configuration
+
+- **`Model.num_classes`** (int, default: 80): Number of detection classes
+- **`Model.bbox_format`** (string, default: "cxcywh"): Bounding box format (`cxcywh`, `xyxy`, or `xywh`)
+- **`Model.output_shape`** (array, optional): Expected model output shape `[batch, detections, features]` (e.g., `[1, 8400, 84]`)
+- **`class_names_path`** (string, optional): Absolute path to text file with class names, one per line (e.g., `/workspaces/deep_ros/deep_object_detection/config/coco_classes.txt`)
+
+### Preprocessing
+
+- **`Preprocessing.input_width`** (int, default: 640): Model input image width
+- **`Preprocessing.input_height`** (int, default: 640): Model input image height
+- **`Preprocessing.normalization_type`** (string, default: "scale_0_1"): Normalization method (`scale_0_1`, `imagenet`, `custom`, `none`)
+- **`Preprocessing.resize_method`** (string, default: "letterbox"): Image resizing method (`letterbox`, `resize`, `crop`, `pad`)
+- **`Preprocessing.color_format`** (string, default: "rgb"): Color format (`rgb` or `bgr`)
+- **`Preprocessing.mean`** (array, default: [0.0, 0.0, 0.0]): Mean values for custom normalization
+- **`Preprocessing.std`** (array, default: [1.0, 1.0, 1.0]): Standard deviation values for custom normalization
+- **`Preprocessing.pad_value`** (int, default: 114): Padding value for letterbox resizing
+
+### Postprocessing
+
+- **`Postprocessing.score_threshold`** (float, default: 0.65): Minimum confidence score
+- **`Postprocessing.nms_iou_threshold`** (float, default: 0.45): IoU threshold for NMS
+- **`Postprocessing.score_activation`** (string, default: "sigmoid"): Score activation (`sigmoid`, `softmax`, `none`)
+- **`Postprocessing.class_score_mode`** (string, default: "all_classes"): Class score mode (`all_classes` or `single_confidence`)
+- **`Postprocessing.enable_nms`** (bool, default: true): Enable non-maximum suppression
+- **`Postprocessing.class_score_start_idx`** (int, default: -1): Start index for class scores (-1 for auto)
+- **`Postprocessing.class_score_count`** (int, default: -1): Number of class scores (-1 for auto)
+
+#### Postprocessing Layout
+
+- **`Postprocessing.layout.batch_dim`** (int, default: 0): Batch dimension index
+- **`Postprocessing.layout.detection_dim`** (int, default: 1): Detection dimension index
+- **`Postprocessing.layout.feature_dim`** (int, default: 2): Feature dimension index
+- **`Postprocessing.layout.bbox_start_idx`** (int, default: 0): Bounding box start index
+- **`Postprocessing.layout.bbox_count`** (int, default: 4): Number of bbox coordinates
+- **`Postprocessing.layout.score_idx`** (int, default: 4): Score index
+- **`Postprocessing.layout.class_idx`** (int, default: 5): Class index
+
+### Backend
+
+- **`Backend.plugin`** (string, required): Backend plugin name (`onnxruntime_cpu` or `onnxruntime_gpu`)
+- **`Backend.execution_provider`** (string, default: "tensorrt"): Execution provider for GPU plugin (`cuda` or `tensorrt`)
+- **`Backend.device_id`** (int, default: 0): GPU device ID (for CUDA/TensorRT)
+- **`Backend.trt_engine_cache_enable`** (bool, default: true): Enable TensorRT engine caching
+- **`Backend.trt_engine_cache_path`** (string, default: "/tmp/deep_ros_ort_trt_cache"): TensorRT engine cache directory
+
+### Input/Output
+
+- **`use_compressed_images`** (bool, default: true): Use compressed images (MultiImageCompressed) vs uncompressed (MultiImage)
+- **`output_detections_topic`** (string, default: "/detections"): Output detections topic name
+
+## Topics
+
+### Key Topics
+
+| Topic | Type | Description |
+|-------|------|-------------|
+| `input_topic` | `deep_msgs/MultiImage` or `deep_msgs/MultiImageCompressed` | Synchronized multi-camera input (compressed or uncompressed) |
+| `output_detections_topic` | `vision_msgs/Detection2DArray` | Detection results (one per image in batch, default: `/detections`) |
+| `/image_annotations` | `visualization_msgs/ImageMarker` | Visualization annotations with bounding boxes (optional) |
+
+**Note:** The node only supports MultiImage/MultiImageCompressed messages. Individual camera topics are not supported.
+
+### Key Services
+
+| Service | Type | Description |
+|---------|------|-------------|
+| `/<node_name>/configure` | `lifecycle_msgs/srv/ChangeState` | Configure the lifecycle node |
+| `/<node_name>/activate` | `lifecycle_msgs/srv/ChangeState` | Activate the lifecycle node |
+| `/<node_name>/deactivate` | `lifecycle_msgs/srv/ChangeState` | Deactivate the lifecycle node |
+| `/<node_name>/cleanup` | `lifecycle_msgs/srv/ChangeState` | Cleanup the lifecycle node |
+| `/<node_name>/shutdown` | `lifecycle_msgs/srv/ChangeState` | Shutdown the lifecycle node |
+| `/<node_name>/get_state` | `lifecycle_msgs/srv/GetState` | Get current lifecycle state |