NVIDIA-accelerated, deep learned semantic image segmentation
Isaac ROS Image Segmentation contains ROS packages for semantic image segmentation.
These packages provide methods for classification of an input image at the pixel level by running GPU-accelerated inference on a DNN model. Each pixel of the input image is predicted to belong to a set of defined classes. The output prediction can be used by perception functions to understand where each class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.
| Package | Model Architecture | Description |
|---|---|---|
| Isaac ROS U-NET | U-NET | Convolutional network popular for biomedical imaging segmentation models |
| Isaac ROS Segformer | Segformer | Transformer-based network that works well for objects of varying scale |
| Isaac ROS Segment Anything | Segment Anything | Segments any object in an image when given a prompt as to which one |
| Isaac ROS Segment Anything2 | Segment Anything2 | Segments and tracks any object in a video stream when given a prompt as to which one |
Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution expected by the DNN model; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image.
Image segmentation provides more information and uses more compute than object detection to produce classifications per pixel, whereas object detection classifies a simpler bounding box rectangle in image coordinates. Object detection is used to know if, and where spatially in a 2D image, the object exists. On the other hand, image segmentation is used to know which pixels belong to the class. One application is using the segmentation result, and fusing it with the corresponding depth information in order to know an object location in a 3D scene.
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
| Sample Graph |
Input Size |
AGX Thor |
x86_64 w/ RTX 5090 |
|---|---|---|---|
| SAM Image Segmentation Graph Full SAM |
720p |
2.23 fps 270 ms @ 30Hz |
20.8 fps 58 ms @ 30Hz |
| SAM Image Segmentation Graph Mobile SAM |
720p |
21.0 fps 580 ms @ 30Hz |
74.1 fps 26 ms @ 30Hz |
| TensorRT Graph PeopleSemSegNet |
544p |
460 fps 15 ms @ 30Hz |
510 fps 12 ms @ 30Hz |
Please visit the Isaac ROS Documentation to learn how to use this repository.
isaac_ros_segformerisaac_ros_segment_anythingisaac_ros_segment_anything2isaac_ros_segment_anything2_interfacesisaac_ros_unet
Update 2025-10-24: Add Segment Anything2 package
