3D Lidar(KITTI) Detection using PointPillars (#1027)

* zhulf0804/PointPillars

* update .gitignore

* update readme to cite zhulf0804/PointPillars repo

* ov extension

This extension will run on CPUs

* pytorch CPP extension

* ov export and e2e inference, test, eval scripts

* Kitti pcd support

* containerised environment setup

* Enable nn model to run on GPU

* cleanup

* moved pointPillars under 3d

* converted xml/bin, extension library still needs to be built

* Remove LICENSE

* Update READMEs

* Sample docker compose yaml

* Third Party license for PointPillars

* Third Party licenses for PointPillars

Author: pj3iL

README.md

@@ -16,6 +16,7 @@ This list gives an overview of all modules available inside the contrib reposito
 * [**OpenVINO Code**](./modules/openvino_code): VSCode extension for AI code completion with OpenVINO.
 * [**Ollama-OpenVINO**](./modules/ollama_openvino): OpenVINO GenAI empowered Ollama which accelerate LLM on Intel platforms(including CPU, iGPU/dGPU, NPU).
 * [**ov_training_kit**](./modules/ov_training_kit): Training Kit Python library -- provides scikit-learn, PyTorch and Tensorflow wrappers for training, optimization, and deployment with OpenVINO on AI PCs.
+* [**3D Point Pillars**](./modules/3d/pointPillars): Use OpenVINO to perform 3D object detection with PointPillars model.
 
 ## How to build OpenVINO with extra modules
 You can build OpenVINO, so it will include the modules from this repository. Contrib modules are under constant development and it is recommended to use them alongside the master branch or latest releases of OpenVINO.
@@ -40,6 +41,7 @@ Additional build instructions are available for the following modules:
 * [**custom_operations**](./modules/custom_operations/README.md)
 * [**ollama_OpenVINO**](./modules/ollama_openvino)
 * [**openvino-langchain**](./modules/openvino-langchain): LangChain.js integrations for OpenVINO™
+* [**3D Point Pillars**](./modules/3d/pointPillars): Check the [README](./modules/3d/pointPillars/README.md) for detailed usage and build instructions.
 
 ## Update the repository documentation
 In order to keep a clean overview containing all contributed modules, the following files need to be created/adapted:

modules/3d/pointPillars/.dockerignore

@@ -0,0 +1,36 @@
+# Git files
+.git
+.gitignore
+.gitattributes
+
+# Python cache
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+.eggs/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# Documentation
+*.md
+!README.md
+figures/
+misc/
+
+# Docker files
+devops/

modules/3d/pointPillars/.gitignore

@@ -0,0 +1,8 @@
+__pycache__
+build/
+*.egg-info
+*.so
+*_logs*
+tmp/
+results/
+.idea

modules/3d/pointPillars/README.md

@@ -0,0 +1,226 @@
+# Note
+
+_This directory is a fork from [PointPillars](https://github.com/zhulf0804/PointPillars) (commit 620e6b0)._
+-----------------------
+
+<details>
+<summary style="font-size:2em; font-weight:600">OpenVino Model Export and Inference</summary>
+
+```bash
+# Install the required packages
+sudo apt update && sudo apt install -y \
+  software-properties-common \
+  build-essential \
+  cmake \
+  git \
+  libx11-6 \
+  libgl1
+```
+
+Tested over python 3.10:
+```bash
+conda install python=3.10
+conda create -n ovpp310 python=3.10
+conda activate ovpp310
+```
+Considering the `REPO_ROOT` variable points to the `pointPillars` directory:
+
+```bash
+REPO_ROOT=/path/to/3d/pointPillars
+```
+
+Install the required python packages:
+```bash
+# Install the required pip packages
+python -m pip install -r "${REPO_ROOT}/requirements-ov.txt"
+```
+
+Build the pytorch extension required for exporting the model:
+```bash
+# Build the pytorch extensions (will be used only to export the model)
+cd ${REPO_ROOT}
+CPU_BUILD=1 python setup.py build_ext --inplace
+cd ..
+```
+
+Build the OpenVino Extension required for inference:
+```bash
+# Build the openvino extension
+cd "${REPO_ROOT}/ov_extensions" && rm -rf build/ && bash build.sh && cd ..
+```
+
+Exporting the model:
+```bash
+# Export the PointPillars .pth model to OpenVINO format
+python "${REPO_ROOT}/export_ov_e2e.py" --checkpoint "${REPO_ROOT}/pretrained/epoch_160.pth" --output "${REPO_ROOT}/pretrained/pointpillars_ov"
+```
+The above command will generate the following five files:
+- pointpillars_ov_config.json
+- pointpillars_ov_nn.bin
+- pointpillars_ov_nn.xml
+- pointpillars_ov_pillar_layer.xml
+- pointpillars_ov_postproc.xml
+
+Running inference using OpenVINO over a single pc data:
+```bash
+# Run inference with OpenVINO
+python "${REPO_ROOT}/e2eOVInference.py" --config "${REPO_ROOT}/pretrained/pointpillars_ov_config.json" --pc_path "${REPO_ROOT}/pointpillars/dataset/demo_data/test/000002.bin"
+```
+The Kitti dataset Download and Preprocess steps are described in [this section](#datasets). Say, it is stored in `${REPO_ROOT}/Datasets`.
+
+Evaluate model over the KITTI val set:
+```bash
+python "${REPO_ROOT}/evaluate-e2eOV.py" --device CPU --data_root "${REPO_ROOT}/Datasets" --config "${REPO_ROOT}/pretrained/pointpillars_full_config.json" --saved_path "${REPO_ROOT}/profiles/evals-e2eov-cpu"
+```
+
+</details>
+
+
+# [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://arxiv.org/abs/1812.05784)
+
+A Simple PointPillars PyTorch Implenmentation for 3D Lidar(KITTI) Detection. [[Zhihu](https://zhuanlan.zhihu.com/p/521277176)]
+
+- It can be run without installing [Spconv](https://github.com/traveller59/spconv), [mmdet](https://github.com/open-mmlab/mmdetection) or [mmdet3d](https://github.com/open-mmlab/mmdetection3d).
+- Only one detection network (PointPillars) was implemented in this repo, so the code may be more easy to read.
+- Sincere thanks for the great open-source architectures [mmcv](https://github.com/open-mmlab/mmcv), [mmdet](https://github.com/open-mmlab/mmdetection) and [mmdet3d](https://github.com/open-mmlab/mmdetection3d), which helps me to learn 3D detetion and implement this repo.
+
+## News
+
+- **2025-02** Making PointPillars a python package out of the code is supported.
+- **2024-04** Exporting PointPillars to ONNX & TensorRT is supported on branch [feature/deployment](https://github.com/zhulf0804/PointPillars/tree/feature/deployment).
+
+    ![](./figures/pytorch_trt.png)
+
+## mAP on KITTI validation set (Easy, Moderate, Hard)
+
+| Repo | Metric | Overall | Pedestrian | Cyclist | Car |
+| :---: | :---: | :---: | :---: | :---: | :---: |
+| this repo | 3D-BBox | 73.3259 62.7834 59.6278 | 51.4642 47.9446 43.8040 | 81.8677 63.6617 60.9126 | 86.6456 76.7439 74.1668 |
+| [mmdet3d v0.18.1](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1) | 3D-BBox  | 72.0537, 60.1114, 55.8320 | 52.0263, 46.4037, 42.4841 | 78.7231, 59.9526, 57.2489 | 85.4118, 73.9780, 67.7630 |
+| this repo | BEV | 77.8540 69.8003 66.6699 | 59.1687 54.3456 50.5023 | 84.4268 67.1409 63.7409 | 89.9664 87.9145 85.7664 |
+| [mmdet3d v0.18.1](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1) | BEV | 76.6485, 67.7609, 64.5605 | 59.0778, 53.3638, 48.4230 | 80.9328, 63.3447, 60.0618 | 89.9348, 86.5743, 85.1967 |
+| this repo | 2D-BBox | 80.5097 74.6120 71.4758 | 64.6249 61.4201 57.5965 | 86.2569 73.0828 70.1726 | 90.6471 89.3330 86.6583 |
+| [mmdet3d v0.18.1](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1) | 2D-BBox | 78.4938, 73.4781, 70.3613 | 62.2413, 58.9157, 55.3660 | 82.6460, 72.3547, 68.4669 | 90.5939, 89.1638, 87.2511 |
+| this repo | AOS | 74.9647 68.1712 65.2817 | 49.3777 46.7284 43.8352 | 85.0412 69.1024 66.2801 | 90.4752 88.6828 85.7298 |
+| [mmdet3d v0.18.1](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1) | AOS | 72.41, 66.23, 63.55 | 46.00, 43.22, 40.94 | 80.85, 67.20, 63.63 | 90.37, 88.27, 86.07 |
+
+- **Note: Here, we report [mmdet3d v0.18.1](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1) (2022/02/09-2022/03/01) performance based on the officially provided [checkpoint](https://github.com/open-mmlab/mmdetection3d/tree/v0.18.1/configs/pointpillars#kitti). Much improvements were made in the [mmdet3d v1.0.0rc1](https://github.com/open-mmlab/mmdetection3d/tree/v1.0.0rc1)**.
+
+## Detection Visualization
+
+![](./figures/pc_pred_000134.png)
+![](./figures/img_3dbbox_000134.png)
+
+## [Install]
+
+Install PointPillars as a python package and all its dependencies as follows:
+
+```
+cd PointPillars/
+pip install -r requirements.txt
+python setup.py build_ext --inplace
+pip install .
+```
+
+## [Datasets]
+
+1. Download
+
+    Download [point cloud](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_velodyne.zip)(29GB), [images](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_image_2.zip)(12 GB), [calibration files](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_calib.zip)(16 MB)和[labels](https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_label_2.zip)(5 MB)。Format the datasets as follows:
+    ```
+    kitti
+        |- training
+            |- calib (#7481 .txt)
+            |- image_2 (#7481 .png)
+            |- label_2 (#7481 .txt)
+            |- velodyne (#7481 .bin)
+        |- testing
+            |- calib (#7518 .txt)
+            |- image_2 (#7518 .png)
+            |- velodyne (#7518 .bin)
+    ```
+
+2. Pre-process KITTI datasets First
+
+    ```
+    cd PointPillars/
+    python pre_process_kitti.py --data_root your_path_to_kitti
+    ```
+
+    Now, we have datasets as follows:
+    ```
+    kitti
+        |- training
+            |- calib (#7481 .txt)
+            |- image_2 (#7481 .png)
+            |- label_2 (#7481 .txt)
+            |- velodyne (#7481 .bin)
+            |- velodyne_reduced (#7481 .bin)
+        |- testing
+            |- calib (#7518 .txt)
+            |- image_2 (#7518 .png)
+            |- velodyne (#7518 .bin)
+            |- velodyne_reduced (#7518 .bin)
+        |- kitti_gt_database (# 19700 .bin)
+        |- kitti_infos_train.pkl
+        |- kitti_infos_val.pkl
+        |- kitti_infos_trainval.pkl
+        |- kitti_infos_test.pkl
+        |- kitti_dbinfos_train.pkl
+    ```
+
+## [Training]
+
+```
+cd PointPillars/
+python train.py --data_root your_path_to_kitti
+```
+
+## [Evaluation]
+
+```
+cd PointPillars/
+python evaluate.py --ckpt pretrained/epoch_160.pth --data_root your_path_to_kitti
+```
+
+## [Test]
+
+```
+cd PointPillars/
+
+# 1. infer and visualize point cloud detection
+python test.py --ckpt pretrained/epoch_160.pth --pc_path your_pc_path
+
+# 2. infer and visualize point cloud detection and gound truth.
+python test.py --ckpt pretrained/epoch_160.pth --pc_path your_pc_path --calib_path your_calib_path  --gt_path your_gt_path
+
+# 3. infer and visualize point cloud & image detection
+python test.py --ckpt pretrained/epoch_160.pth --pc_path your_pc_path --calib_path your_calib_path --img_path your_img_path
+
+
+e.g.
+a. [infer on val set 000134]
+
+python test.py --ckpt pretrained/epoch_160.pth --pc_path pointpillars/dataset/demo_data/val/000134.bin
+
+or
+
+python test.py --ckpt pretrained/epoch_160.pth --pc_path pointpillars/dataset/demo_data/val/000134.bin \
+               --calib_path pointpillars/dataset/demo_data/val/000134.txt \
+               --img_path pointpillars/dataset/demo_data/val/000134.png \
+               --gt_path pointpillars/dataset/demo_data/val/000134_gt.txt
+
+b. [infer on test set 000002]
+
+python test.py --ckpt pretrained/epoch_160.pth --pc_path pointpillars/dataset/demo_data/test/000002.bin
+
+or
+
+python test.py --ckpt pretrained/epoch_160.pth --pc_path pointpillars/dataset/demo_data/test/000002.bin \
+               --calib_path pointpillars/dataset/demo_data/test/000002.txt \
+               --img_path pointpillars/dataset/demo_data/test/000002.png
+```
+
+## Acknowledements
+
+Thanks for the open source code [mmcv](https://github.com/open-mmlab/mmcv), [mmdet](https://github.com/open-mmlab/mmdetection) and [mmdet3d](https://github.com/open-mmlab/mmdetection3d).

modules/3d/pointPillars/devops/Dockerfile

@@ -0,0 +1,84 @@
+ARG PYTHON_VERSION=3.10
+ARG UBUNTU_VERSION=24.04
+
+ARG http_proxy
+ARG https_proxy
+ARG no_proxy
+
+FROM ubuntu:${UBUNTU_VERSION} AS base-selected
+
+# Set environment variables
+ENV DEBIAN_FRONTEND=noninteractive
+ENV LANG=C.UTF-8
+ENV LC_ALL=C.UTF-8
+
+# Install system dependencies
+RUN apt update && \
+    apt install -y --no-install-recommends \
+    software-properties-common \
+    cmake \
+    build-essential \
+    libx11-6 \
+    libgl1
+
+# Install Python and pip
+ARG PYTHON_VERSION
+RUN add-apt-repository -y ppa:deadsnakes/ppa
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    python${PYTHON_VERSION} \
+    python${PYTHON_VERSION}-dev \
+    python${PYTHON_VERSION}-distutils \
+    python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+
+# Create symbolic links for python
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 1 && \
+    update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1
+
+# Upgrade pip, setuptools, wheel, blinker
+RUN python -m pip install --no-cache-dir --upgrade --ignore-installed pip setuptools wheel blinker
+
+# intel GPU
+RUN cd /usr/lib/python3/dist-packages/ && \
+    ln -s apt_pkg.cpython-312-x86_64-linux-gnu.so apt_pkg.so
+
+RUN add-apt-repository -y ppa:kobuk-team/intel-graphics || true
+RUN apt update && \
+    apt install -y --no-install-recommends \
+    libze-intel-gpu1 \
+    intel-opencl-icd
+
+
+
+
+# Set working directory
+WORKDIR /workspace
+
+# Install Python packages
+COPY requirements.txt .
+RUN python -m pip install --no-cache-dir -r requirements.txt
+
+# Building torch extensions...
+COPY pointpillars/ ./pointpillars/
+COPY setup.py .
+RUN CPU_BUILD=1 python setup.py build_ext --inplace
+
+# Building OpenVINO extensions...
+COPY ov_extensions ./ov_extensions
+RUN cd ov_extensions && rm -rf build && bash build.sh && cd ..
+
+# Exporting OpenVINO model
+COPY export_ov_e2e.py .
+COPY pretrained ./pretrained
+# python export_ov_e2e.py --checkpoint pretrained/epoch_160.pth --output pretrained/pointpillars_ov
+
+# Test
+COPY e2eOVInference.py .
+COPY test-e2eOV.py .
+# python test-e2eOV.py --device CPU
+# python test-e2eOV.py --device GPU
+
+# Evaluation Kitti
+COPY evaluate-e2eOV.py .
+# python evaluate-e2eOV.py --data_root /Datasets --nsamples 10