Add Robotics AI Suite. (open-edge-platform#167)

Author: jouillet

.github/CODEOWNERS

@@ -9,4 +9,5 @@ SECURITY.md @xwu2intel
 /metro-ai-suite/loitering-detection/ @vagheshp @tjanczak @xwu2intel
 /manufacturing-ai-suite/pallet-defect-detection/ @ajagadi1 @sugnanprabhu @rrajore
 /manufacturing-ai-suite/weld-porosity/ @ajagadi1 @sugnanprabhu @rrajore
-/manufacturing-ai-suite/wind-turbine-anomaly-detection/ @vkb1 @sathyendranv @pooja-intel
+/manufacturing-ai-suite/wind-turbine-anomaly-detection/ @vkb1 @sathyendranv @pooja-intel
+/robotics-ai-suite/ @jouillet @jb-balaji @pirouf @mohitmeh12

README.md

@@ -19,6 +19,9 @@ These suites simplify the creation of custom AI solutions for specific industrie
 
 [The Media & Entertainment AI Suite](media-and-entertainment-ai-suite) provides libraries and sample applications to accelerate solution development for high-performance, high-quality, live video production helping improve viewer experience.
 
+[The Robotics AI Suite](robotics-ai-suite) provides ready to use samples leveraging AI to help solve common robotics problems such as perception, navigation, simulation, and planning.
+
+
 The Edge AI Suites project hosts a collection of sample applications organized as follows:
 
 | Suite | Sample Application | Get Started | Developers Docs |
@@ -33,6 +36,8 @@ The Edge AI Suites project hosts a collection of sample applications organized a
 | Manufacturing AI Suite | [Wind Turbine Anomaly Detection](manufacturing-ai-suite/wind-turbine-anomaly-detection/) | [Link](manufacturing-ai-suite/wind-turbine-anomaly-detection/docs/user-guide/get-started.md) | [Docker deployment](manufacturing-ai-suite/wind-turbine-anomaly-detection/docs/user-guide/get-started.md#deploy-with-docker-compose-single-node) and [Helm deployment](manufacturing-ai-suite/wind-turbine-anomaly-detection/docs/user-guide/how-to-deploy-with-helm.md) |
 | Retail AI Suite | [Automated Self Checkout](https://github.com/intel-retail/automated-self-checkout) | [Link](https://github.com/intel-retail/automated-self-checkout?tab=readme-ov-file#-quickstart) | [Advanced Guide](https://intel-retail.github.io/documentation/use-cases/automated-self-checkout/automated-self-checkout.html) |
 | Retail AI Suite | [Loss Prevention](https://github.com/intel-retail/loss-prevention) | [Link](https://github.com/intel-retail/loss-prevention?tab=readme-ov-file#quickstart) | [Advanced Guide](https://intel-retail.github.io/documentation/use-cases/loss-prevention/loss-prevention.html) |
+| Robotics AI Suite | [ACT Sample](robotics-ai-suite/act-sample) | [Link](robotics-ai-suite/act-sample) | [Tutorial](robotics-ai-suite/act-sample/README.md) |
+| Robotics AI Suite | [ORB-SLAM3 Sample](robotics-ai-suite/orb-slam3-sample) | [Link](robotics-ai-suite/orb-slam3-sample) | [Tutorial](robotics-ai-suite/orb-slam3-sample/README.md) |
 
 Please visit each sample application sub-directory for respective **Getting Started**, **Customization** instructions.
 
@@ -54,4 +59,4 @@ The **Edge AI Suites** project is licensed under the [APACHE 2.0](LICENSE), exce
 |[Holographic Sensor Fusion](metro-ai-suite/holographic-sensor-fusion) | [LIMITED EDGE SOFTWARE DISTRIBUTION LICENSE AGREEMENT](metro-ai-suite/holographic-sensor-fusion/LICENSE.txt) |
 
 Last Updated Date: May 30, 2025.  
- 
+ 

robotics-ai-suite/README.assets/sdk_architecture.png

@@ -0,0 +1,57 @@
+# Robotics AI Suite
+
+The [Robotics AI Suite](https://eci.intel.com/embodied-sdk-docs/content/Intel_embodied_Intelligence_SDK.html) is a collection of intuitive, easy-to-use software stack designed to streamline the development process of Embodied Intelligence product and applications on Intel platform. The suite provides developers with a comprehensive environment for developing, testing, and optimizing Embodied Intelligence software and algorithms efficiently. It also provides necessary software framework, libraries, tools, Best known configuration(BKC), tutorials and example codes to facilitate AI solution development.
+
+The Robotics AI Suite includes the below features:
+
+- Comprehensive software platform from BSP, acceleration libraries, SDK to reference demos, with documentation and developer tutorials
+- Real-time BKC, Linux RT kernel and optimized EtherCAT
+- Traditional vision and motion planning acceleration on CPU, Reinforcement/Imitation Learning-based manipulation, AI-based vision & LLM/VLM acceleration on iGPU & NPU
+- Typical workflows and examples including ACT/DP-based manipulation, LLM task planning, Pick & Place, ORB-SLAM3, etc
+
+![architecture](README.assets/sdk_architecture.png)
+
+## Description
+
+This software architecture is designed to power Embodied Intelligence systems by integrating computer vision, AI-driven manipulation, locomotion, SLAM, and large models into a unified framework. Built on ROS2 middleware, it takes advantage of Intel’s CPU, iGPU, dGPU, and NPU to optimize performance for robotics and AI applications. The stack includes high-performance AI frameworks, real-time libraries, and system-level optimizations, making it a comprehensive solution for Embodied Intelligence products.
+
+At the highest level, the architecture is structured around key reference pipelines and demos that demonstrate its core capabilities. These include Vision Servo, which enhances robotic perception using AI-powered vision modules, and ACT-based Manipulation, which applies reinforcement learning and imitation learning to improve robotic grasping and movement. Optimized Locomotion leverages traditional control algorithms like MPC (Model Predictive Control) and LQR (Linear Quadratic Regulator), alongside reinforcement learning models for adaptive motion. Additionally, the ORB-SLAM3 pipeline focuses on real-time simultaneous localization and mapping, while LLM Task Planning integrates large language models for intelligent task execution.
+
+Beneath these pipelines, the software stack includes specialized AI and robotics modules. The vision module supports CNN-based models, OpenCV, and PCL operators for optimized perception, enabling robots to interpret their surroundings efficiently. The manipulation module combines traditional motion planning with AI-driven control, allowing robots to execute complex movements. For locomotion, the system blends classic control techniques with reinforcement learning models, ensuring smooth and adaptive movement. Meanwhile, SLAM components such as GPU ORB extraction and ADBSCAN optimization enhance mapping accuracy, and BEV (Bird’s Eye View) models contribute to improved spatial awareness. The large model module supports LLMs, Vision-Language Models (VLM), and Vision-Language-Action Models (VLA), enabling advanced reasoning and decision-making capabilities.
+
+At the core of the system is ROS2 middleware and acceleration frameworks, which provide a standardized framework for robotics development. The architecture is further enhanced by Intel’s AI acceleration libraries, including Intel® OpenVINO™ for deep learning inference, Intel® LLM Library for PyTorch* (IPEX-LLM) for optimized large model execution, and compatibility with TensorFlow*, PyTorch*, and ONNX*. The Intel® oneAPI compiler and libraries offer high-performance computing capabilities, leveraging oneMKL for mathematical operations, oneDNN for deep learning, and oneTBB for parallel processing. Additionally, Intel’s real-time libraries ensure low-latency execution, with tools for performance tuning and EtherCAT-based industrial communication.
+
+To ensure seamless integration with robotic hardware, the suite runs on a real-time optimized Linux BSP. It includes support for optimized EtherCAT and camera drivers, along with Intel-specific features such as Speed Shift Technology and Cache Allocation to enhance power efficiency and performance. These system-level enhancements allow the software stack to deliver high responsiveness, making it suitable for real-time robotics applications.
+
+Overall, the Robotics AI Suite provides a highly optimized, AI-driven framework for robotics and Embodied Intelligence, combining computer vision, motion planning, real-time processing, and large-scale AI models into a cohesive system. By leveraging Intel’s hardware acceleration and software ecosystem, it enables next-generation robotic applications with enhanced intelligence, efficiency, and adaptability.
+
+## Collection
+
+**AI Suite Pipelines:**
+
+| Pipeline Name | Description |
+| ------------- | ----------- |
+| [Imitation Learning - ACT](orb-slam3-sample) | Imitation learning pipeline using Action Chunking with Transformers(ACT) algorithm to train and evaluate in simulator or real robot environment with Intel optimization |
+| [VSLAM: ORB-SLAM3](act-sample) | One of popular real-time feature-based SLAM libraries able to perform Visual, Visual-Inertial and Multi-Map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models |
+
+**Intel® OpenVINO™ optimized model algorithms:**
+
+| Algorithm | Description |
+| --------- | ----------- |
+| [YOLOv8](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | CNN based object detection |
+| [YOLOv12](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | CNN based object detection |
+| [MobileNetV2](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | CNN based object detection |
+| [SAM](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Transformer based segmentation |
+| [SAM2](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Extend SAM to video segmentation and object tracking with cross attention to memory |
+| [FastSAM](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Lightweight substitute to SAM |
+| [MobileSAM](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Lightweight substitute to SAM (Same model architecture with SAM. Can refer to OpenVINO SAM tutorials for model export and application) |
+| [U-NET](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | CNN based segmentation and diffusion model |
+| [DETR](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Transformer based object detection |
+| [DETR GroundingDino](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Transformer based object detection |
+| [CLIP](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials.html#model-tutorials) | Transformer based image classification |
+| [Action Chunking with Transformers - ACT](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials/model_act.html#model-act) | An end-to-end imitation learning model designed for fine manipulation tasks in robotics |
+| [Feature Extraction Model: SuperPoint](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials/model_superpoint.html#model-superpoint) | A self-supervised framework for interest point detection and description in images, suitable for a large number of multiple-view geometry problems in computer vision |
+| [Feature Tracking Model: LightGlue](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials/model_lightglue.html#model-lightglue) | A model designed for efficient and accurate feature matching in computer vision tasks |
+| [Bird’s Eye View Perception: Fast-BEV](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials/model_fastbev.html#model-fastbev) | Obtaining a BEV perception is to gain a comprehensive understanding of the spatial layout and relationships between objects in a scene |
+| [Monocular Depth Estimation: Depth Anything V2](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/model_tutorials/model_depthanythingv2.html#model-depthanythingv2) | A powerful tool that leverages deep learning to infer 3D information from 2D images |
+

robotics-ai-suite/README.md

@@ -0,0 +1,179 @@
+# Imitation Learning - ACT
+
+Imitation learning is a machine learning approach where a model is trained to mimic expert behavior by observing and replicating demonstrations, enabling it to perform tasks similarly to the expert. ACT is an action chunking policy with Transformers, an architecture designed for sequence modeling, and train it as a conditional VAE (CVAE) to capture the variability in human data. It significantly outperforms previous imitation learning algorithms on a range of simulated and real-world fine manipulation tasks.
+
+We have built an imitation learning pipeline for ACT, which can be used to train and evaluate the ACT model on different tasks both in simulation and real robot environment. In this sample pipeline, we provided source code optimized by Intel® Extension for PyTorch and Intel® OpenVINO™ to accelerate the process.
+
+In this tutorial, we will introduce how to setup ACT pipeline.
+
+## Prerequisites
+
+Please make sure you have finished setup steps in [Installation & Setup](https://eci.intel.com/embodied-sdk-docs/content/installation_setup.html) and followed refer to [oneAPI doc](https://eci.intel.com/embodied-sdk-docs/content/developer_tools_tutorials/oneapi.html#oneapi-install-label) to setup Intel® oneAPI packages.
+
+## Installation
+
+### ALOHA real robot environment setup (Optional)
+
+Follow the [stationary ALOHA guide](https://docs.trossenrobotics.com/aloha_docs/2.0/getting_started/stationary.html) to build real robot platform.
+
+### Virtual environment setup
+
+1. Create a Python 3.10 virtual environment with the following command:
+
+```
+$ sudo apt install python3-venv
+$ python3 -m venv act
+```
+
+2. Activate the virtual environment with the following command:
+
+```
+$ source act/bin/activate
+```
+
+### Install Intel® Extension for PyTorch
+
+> [!IMPORTANT]
+> Intel® Extension for PyTorch workloads are incompatible with the NPU driver. For more details, please refer to the [Troubleshooting page](https://eci.intel.com/embodied-sdk-docs/content/troubleshooting.html).
+
+Install the Intel® Extension for PyTorch with the following command:
+
+```
+$ pip install torch==2.3.1+cxx11.abi torchvision==0.18.1+cxx11.abi torchaudio==2.3.1+cxx11.abi intel-extension-for-pytorch==2.3.110+xpu oneccl_bind_pt==2.3.100+xpu ipex-llm==2.2.0b20241224 --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
+```
+
+### Install Intel® OpenVINO™
+
+Install the Intel® OpenVINO™ with the following command:
+
+```
+$ pip install openvino==2024.6.0
+```
+
+### Dependencies setup
+
+Install the dependencies with the following command:
+
+```
+$ pip install pyquaternion==0.9.9 pyyaml==6.0 rospkg==1.5.0 pexpect==4.8.0 mujoco==3.2.6 dm_control==1.0.26 matplotlib==3.10.0 einops==0.6.0 packaging==23.0 h5py==3.12.1 ipython==8.12.0 opencv-python==4.10.0.84 transformers==4.37.0 accelerate==0.23.0 bigdl-core-xe-21==2.6.0b2 bigdl-core-xe-addons-21==2.6.0b2 bigdl-core-xe-batch-21==2.6.0b2 huggingface-hub==0.24.7
+```
+
+### Install ACT package
+
+The Embodied Intelligence SDK provides optimized source code for Intel® Extension for PyTorch and Intel® OpenVINO™. To get the source code with the following command:
+
+For Intel® Extension for PyTorch:
+
+```
+$ sudo apt install act-ipex 
+$ sudo chown -R $USER /opt/act-ipex/
+```
+
+For Intel® OpenVINO™:
+
+```
+$ sudo apt install act-ov
+$ sudo chown -R $USER /opt/act-ov/
+```
+
+### Install DETR
+
+Install the DETR with the following command:
+
+```
+$ cd <path_to_act>/detr/
+$ pip install -e .
+```
+
+## Run pipeline
+
+### Inference
+
+1. You can download our pre-trained weights from this link: [Download Link](https://eci.intel.com/embodied-sdk-docs/_downloads/sim_insertion_scripted.zip). The command of training is the same as above, but you need to set the argument `--ckpt_dir` to the path of the pre-trained weights.
+
+2. Convert the model checkpoint to OpenVINO IR **(Optional)**
+
+`ov_convert.py` is a script provided to convert the PyTorch model to OpenVINO IR. You can find the script in the `act-ov` directory, and see the usage with the following command:
+
+```
+$ cd /opt/act-ov/
+$ python3 ov_convert.py -h
+```
+
+For example, you can convert the model with the following command:
+
+```
+$ python3 ov_convert.py --ckpt_path <your_ckpt_path> --height 480 --weight 640 --camera_num 4 --chunk_size 100
+```
+
+> [!IMPORTANT]
+> Please make sure the arguments `--chunk_size`, `--kl_weight`, `--hidden_dim`, `--dim_feedforward`, `--camera_num` are the same as the training arguments.
+
+3. The pipeline supports configurations with up to four cameras. You can modify the `constants.py` file in the source directory to define the number of cameras. Below are examples of configurations for four cameras and one camera:
+
+```
+# In /opt/act-ov/constants.py
+SIM_TASK_CONFIGS = {
+    'sim_insertion_scripted': {
+        'dataset_dir': DATA_DIR + '/sim_insertion_scripted',
+        'num_episodes': 50,
+        'episode_len': 400,
+        'camera_names': ['top', 'angle', 'left_wrist', 'right_wrist']
+    },
+}
+
+# In /opt/act-ipex/constants.py
+SIM_TASK_CONFIGS = {
+    'sim_insertion_scripted': {
+        'dataset_dir': DATA_DIR + '/sim_insertion_scripted',
+        'num_episodes': 50,
+        'episode_len': 400,
+        'camera_names': ['top']
+    },
+}
+```
+
+Below is a camera viewer showcasing four different camera perspectives, the left one is the `angle` camera, and the right one is the `top` camera. The middle two are the `left and right wrist` cameras, respectively.
+
+![act-sim-cameras](README.assets/act-sim-cameras.png)
+
+4. Evaluate the policy with the following command:
+
+```
+$ python3 imitate_episodes.py --task_name sim_insertion_scripted --ckpt_dir <ckpt dir> --policy_class ACT --kl_weight 10 --chunk_size 100 --hidden_dim 512 --batch_size 8 --dim_feedforward 3200 --num_epochs 2000  --lr 1e-5 --seed 0 --device GPU --eval
+```
+
+> [!NOTE]
+> `--eval` is used to evaluate the policy.
+> `--device` is used to set the device to CPU or GPU.
+> `--temporal_agg` can be used to enable the temporal aggregation algorithm.
+> `--onscreen_render` can be used to enable onscreen rendering.
+
+When the `--onscreen_render` parameter is enabled, the successful inference result appears as follows:
+
+![act-sim-insertion-demo](README.assets/act-sim-insertion-demo.gif)
+
+### Training **(Optional)**
+
+> [!IMPORTANT]
+> Please refer to the [ALOHA paper](https://arxiv.org/abs/2304.13705) for instructions on setting up a machine with the training environment.
+
+1. Generate 50 episodes with the following command:
+
+```
+# Bimanual Insertion task
+$ python3 record_sim_episodes.py --task_name sim_insertion_scripted --dataset_dir <data save dir> --num_episodes 50
+```
+
+2. Visualize the episode with the following command:
+
+```
+$ python3 visualize_episodes.py --dataset_dir <data save dir> --episode_idx 0
+```
+
+3. Train ACT with the following command:
+
+```
+# Bimanual Insertion task
+$ python3 imitate_episodes.py --task_name sim_insertion_scripted --ckpt_dir <ckpt dir> --policy_class ACT --kl_weight 10 --chunk_size 100 --hidden_dim 512 --batch_size 8 --dim_feedforward 3200 --num_epochs 2000  --lr 1e-5 --seed 0
+```

robotics-ai-suite/act-sample/README.assets/act-sim-cameras.png

@@ -0,0 +1,9 @@
+# Patches for Action Chunking with Transformers (ACT)
+
+The following patches are provided to enhance the ACT source available at: https://github.com/tonyzhaozh/act
+
+| Directory    | Enhancement                  |
+| ------------ | ---------------------------- |
+| [ipex](ipex) | Intel® Extension for PyTorch |
+| [ov](ov)     | Intel® OpenVINO™             |
+