Skip to content

Commit 70be6c5

Browse files
[DOCS] Robotics Suite Jazzy Tabs - Embodied Docs (#1146)
Co-authored-by: Sebastian Golebiewski <[email protected]>
1 parent 1719f59 commit 70be6c5

File tree

13 files changed

+1171
-1007
lines changed

13 files changed

+1171
-1007
lines changed
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Model Tutorials
2+
3+
The OpenVINO™ toolkit supports most TensorFlow and PyTorch models. The following table lists deep-learning models commonly used in the Embodied Intelligence solutions, and information on how to run them on Intel® platforms:
4+
5+
| Algorithm | Description | Link |
6+
|---------------|----------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
7+
| YOLOv8 | CNN-based object detection | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov8-optimization |
8+
| YOLOv12 | CNN-based object detection | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov12-optimization |
9+
| MobileNetV2 | CNN-based object detection | https://github.com/openvinotoolkit/open_model_zoo/blob/master/models/public/mobilenet-v2-1.0-224 |
10+
| SAM | Transformer-based segmentation | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/segment-anything |
11+
| SAM2 | Extend SAM to video segmentation and object tracking with cross attention to memory | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/sam2-image-segmentation |
12+
| FastSAM | Lightweight substitute to SAM | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/fast-segment-anything |
13+
| MobileSAM | Lightweight substitute to SAM (Same model architecture as SAM. See OpenVINO SAM tutorials for model export and application | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/segment-anything |
14+
| U-NET | CNN-based segmentation and diffusion model | https://community.intel.com/t5/Blogs/Products-and-Solutions/Healthcare/Optimizing-Brain-Tumor-Segmentation-BTS-U-Net-model-using-Intel/post/1399037?wapkw=U-Net |
15+
| DETR | Transformer-based object detection | https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/detr-resnet50 |
16+
| GroundingDino | Transformer-based object detection | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/grounded-segment-anything |
17+
| CLIP | Transformer-based image classification | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/clip-zero-shot-image-classification |
18+
| Qwen2.5VL | Multimodal large language model | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/qwen2.5-vl |
19+
| Whisper | Automatic speech recognition | https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/whisper-asr-genai |
20+
| FunASR | Automatic speech recognition | See the FunASR Setup ****funasr-setup**** in LLM Robotics sample pipeline |
21+
22+
23+
> **Attention:**
24+
When following these tutorials for model conversion, ensure that the OpenVINO toolkit version used for model conversion is the same as the runtime version used for inference. Otherwise, unexpected errors may occur, especially if the model is converted using a newer version and the runtime is an older version. See details in the [Troubleshooting](../troubleshooting.rst) section.
25+
26+
Please also find information for the models of imitation learning, grasp generation, simultaneous localization and mapping (SLAM) and bird's-eye view (BEV):
27+
28+
> **Note:**
29+
Before using these models, read the [AI Content Disclaimer](../../legal.rst).
30+
31+
<!--hide_directive:::{toctree}
32+
:maxdepth: 1
33+
34+
model_tutorials/model_act
35+
model_tutorials/model_cns
36+
model_tutorials/model_dp
37+
model_tutorials/model_idp3
38+
model_tutorials/model_superpoint
39+
model_tutorials/model_lightglue
40+
model_tutorials/model_fastbev
41+
model_tutorials/model_depthanythingv2
42+
model_tutorials/model_rdt
43+
:::hide_directive-->

robotics-ai-suite/docs/embodied/developer_tools_tutorials/model_tutorials.rst

Lines changed: 0 additions & 80 deletions
This file was deleted.

robotics-ai-suite/docs/embodied/fragment_packages-jammy.rst

Lines changed: 11 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,21 @@
1+
2+
Packages
3+
########
4+
15
.. list-table::
26
:widths: 20 40 50
37
:header-rows: 1
48

59
* - Component Group
610
- Package
711
- Description
8-
* - :ref:`LinuxBSP <linuxbsp>`
12+
* - :ref:`Linux Board Support Package (BSP) <linuxbsp>`
913
- | linux-intel-rt-experimental
1014
| linux-intel-experimental
11-
- Intel's Linux LTS real-time kernel (preempt-rt) and generic kernel, kernel version is 6.12.8
15+
- Linux LTS real-time kernel (preempt-rt) optimized for Intel® platforms, version 6.12 and generic kernel, version 6.12.8
1216
* - `Linux Runtime Optimization <https://eci.intel.com/docs/3.3/appendix.html#eci-kernel-boot-optimizations>`__
1317
- customizations-grub
14-
- Linux ECI and Intel's GRUB Customization
18+
- Linux environment for Edge Controls for Industrial (ECI) and Intel-customized GRUB
1519
* - `Linux firmware <https://eci.intel.com/docs/3.3/development/tutorials/enable-graphics.html>`__
1620
- linux-firmware
1721
- Linux firmware with Ultra iGPU firmware
@@ -23,7 +27,7 @@
2327
| ighethercat-dpdk-examples
2428
| ecat-enablekit
2529
| ecat-enablekit-dpdk
26-
- Optimized open source IgH EtherCAT Master Stack, it supports on kernel space and user space
30+
- Optimized open-source IgH EtherCAT Master Stack for kernel space or user space
2731
* - `Motion Control Gateway <https://eci.intel.com/docs/3.3/development/tutorials/enable-ros2-motion-ctrl-gw.html>`__
2832
- | rt-data-agent
2933
| ros-humble-agvm
@@ -51,13 +55,13 @@
5155
| ros-humble-jaka-moveit-py
5256
| ros-humble-run-jaka-moveit
5357
| ros-humble-run-jaka-plc
54-
- The Industrial Motion-Control ROS2 Gateway is the communication bridge between DDS/RSTP wire-protocol ROS2 implementation and Motion Control (MC) IEC-61131-3 standard Intel implementation
58+
- The Industrial Motion-Control ROS2 Gateway is the communication bridge between the DDS and RSTP wire-protocol ROS2 implementation, and the Motion Control (MC) IEC-61131-3 standard Intel implementation.
5559
* - :doc:`VSLAM: ORB-SLAM3 <sample_pipelines/ORB_VSLAM>`
5660
- | libpangolin
5761
| liborb-slam3
5862
| liborb-slam3-dev
5963
| orb-slam3
60-
- Visual SLAM demo pipeline based on ORB-SLAM3. Refer to :doc:`VSLAM: ORB-SLAM3 <sample_pipelines/ORB_VSLAM>` for installation and launching tutorials.
64+
- Visual SLAM demo pipeline based on ORB-SLAM3. See :doc:`VSLAM: ORB-SLAM3 <sample_pipelines/ORB_VSLAM>` for installation and launch tutorials.
6165
* - `RealSense Camera <https://wiki.ros.org/RealSense>`__
6266
- | librealsense2
6367
| librealsense2-dev
@@ -69,7 +73,7 @@
6973
| ros-humble-realsense2-camera
7074
| ros-humble-realsense2-camera-msgs
7175
| ros-humble-realsense2-description
72-
- RealSense Camera driver and tools.
76+
- RealSense camera's driver and tools.
7377
* - :doc:`Imitation Learning - ACT <sample_pipelines/imitation_learning_act>`
7478
- | act-ov
7579
- Action Chunking with Transformers (ACT), a method that trains a generative model to understand and predict action sequences.
@@ -83,10 +87,3 @@
8387
* - :doc:`Robotics Diffusion Transformer (RDT) <sample_pipelines/robotics_diffusion_transformer>`
8488
- | rdt-ov
8589
- Robotics Diffusion Transformer (RDT), the largest bimanual manipulation foundation model with strong generalizability.
86-
87-
.. toctree::
88-
:maxdepth: 1
89-
:hidden:
90-
91-
packages/linuxbsp
92-
packages/mc_gateway
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Install Intel® RealSense™ SDK
2+
3+
Intel® RealSense™ SDK 2.0 is a cross-platform library for Intel® RealSense™ depth cameras.
4+
The SDK allows depth and color streaming, and provides intrinsic and extrinsic calibration information.
5+
The library also offers synthetic streams (pointcloud, depth aligned to color and vise-versa), and a built-in
6+
support for record and playback of streaming sessions.
7+
8+
Intel® RealSense™ SDK 2.0 supports Robot Operating System (ROS) and ROS 2, allowing you to access commonly used robotic functionality with ease.
9+
10+
ROS is a set of open-source software libraries and tools that help you build robot applications. For more information, see https://www.ros.org/.
11+
12+
## Installation
13+
14+
1. Register the server's public key:
15+
16+
```bash
17+
18+
sudo mkdir -p /etc/apt/keyrings
19+
curl -sSf https://librealsense.intel.com/Debian/librealsense.pgp | sudo tee /etc/apt/keyrings/librealsense.pgp > /dev/null
20+
```
21+
22+
2. Ensure that apt HTTPS support is installed:
23+
24+
```bash
25+
26+
sudo apt-get install apt-transport-https
27+
```
28+
29+
3. Add the server to the list of repositories:
30+
31+
```bash
32+
33+
echo "deb [signed-by=/etc/apt/keyrings/librealsense.pgp] https://librealsense.intel.com/Debian/apt-repo `lsb_release -cs` main" | \
34+
sudo tee /etc/apt/sources.list.d/librealsense.list
35+
```
36+
37+
4. Update your apt repository caches after setting up the repositories:
38+
39+
```bash
40+
41+
sudo apt update
42+
```
43+
44+
5. Install the RealSense drivers and libraries:
45+
46+
```bash
47+
sudo apt install librealsense2-dkms
48+
sudo apt install librealsense2=2.55.1-0~realsense.12474
49+
```
50+
51+
> **Attention:**
52+
The command above has fixed the ``librealsense2`` package version; therefore, you need to install dependent packages, for example ``librealsense2-utils``, ``librealsense2-dev``, and ``librealsense2-gl``.
53+
54+
6. (Optional) Install the ROS wrappers for Intel RealSense depth cameras:
55+
56+
<!--hide_directive::::{tab-set}hide_directive-->
57+
<!--hide_directive:::{tab-item}hide_directive--> **Jazzy**
58+
<!--hide_directive:sync: tab1 hide_directive-->
59+
60+
```bash
61+
sudo apt install ros-jazzy-realsense2-camera
62+
```
63+
64+
<!--hide_directive:::hide_directive-->
65+
<!--hide_directive:::{tab-item}hide_directive--> **Humble**
66+
<!--hide_directive:sync: tab2 hide_directive-->
67+
68+
```bash
69+
sudo apt install ros-humble-realsense2-camera
70+
```
71+
72+
<!--hide_directive:::hide_directive-->
73+
<!--hide_directive::::hide_directive-->
74+
75+
#. (Optional) Install other tools or packages of Intel RealSense depth cameras:
76+
77+
See the [installation link](https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md)>
78+
to install librealsense packages and more other tools from Intel® RealSense™ depth camera sources.
79+
80+
## Troubleshooting
81+
82+
### Errors for ``librealsense2-dkms`` installation
83+
84+
If errors occur during ``librealsense2-dkms`` package installation, you have options to fix it:
85+
86+
- Install librealsense SDK by using original Linux drivers.
87+
88+
Once errors occur during installing ``librealsenses2-dkms`` package, it probably is introduced by unmatched kernel version. You can try the following workaround:
89+
90+
```bash
91+
92+
sudo rm -rf /var/lib/dpkg/info/librealsense2-dkms*
93+
sudo apt install librealsense2-dkms
94+
```
95+
96+
- If the option above doesn't work , try to build and install from the source code.
97+
98+
Follow the steps in `the link <https://github.com/IntelRealSense/librealsense/blob/development/doc/installation.md>`_ to download the librealsense source code and build it.
99+
100+
### Errors for unmet dependencies
101+
102+
If you encounter unmet dependencies during the installation of ROS wrappers for Intel RealSense depth cameras, for example:
103+
104+
```shell
105+
106+
The following packages have unmet dependencies:
107+
ros-humble-librealsense2-tools : Depends: ros-humble-librealsense2 (= 2.55.1-1eci9) but 2.55.1-1jammy.20241125.233100 is to be installed
108+
E: Unable to correct problems, you have held broken packages.
109+
```
110+
111+
This issue is probably caused by the mismatched versions of the ROS wrapper and the librealsense2 package.
112+
You can try to fix it by specifying the version of the dependent package. You can try the following for the example given above:
113+
114+
<!--hide_directive::::{tab-set}hide_directive-->
115+
<!--hide_directive:::{tab-item} hide_directive--> **Jazzy**
116+
<!--hide_directive:sync: tab1 hide_directive-->
117+
118+
```bash
119+
sudo apt install ros-jazzy-librealsense2=2.55.1-1eci9
120+
sudo apt install ros-jazzy-librealsense2-tools=2.55.1-1eci9
121+
```
122+
123+
<!--hide_directive:::hide_directive-->
124+
<!--hide_directive:::{tab-item}hide_directive--> **Humble**
125+
<!--hide_directive:sync: tab2 hide_directive-->
126+
127+
```bash
128+
sudo apt install ros-humble-librealsense2=2.55.1-1eci9
129+
sudo apt install ros-humble-librealsense2-tools=2.55.1-1eci9
130+
```
131+
132+
<!--hide_directive:::hide_directive-->
133+
<!--hide_directive::::hide_directive-->

0 commit comments

Comments
 (0)