Lidar AI Solution

This is a highly optimized solution for self-driving 3D-lidar repository. It does a great job of speeding up sparse convolution/CenterPoint/BEVFusion/OSD/Conversion.

Pipeline overview

GetStart

$ git clone --recursive https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution
$ cd Lidar_AI_Solution

• For each specific task please refer to the readme in the sub-folder.

3D Sparse Convolution

A tiny inference engine for 3d sparse convolutional networks using int8/fp16.

• Tiny Engine: Tiny Lidar-Backbone inference engine independent of TensorRT.
• Flexible: Build execution graph from ONNX.
• Easy To Use: Simple interface and onnx export solution.
• High Fidelity: Low accuracy drop on nuScenes validation.
• Low Memory: 422MB@SCN FP16, 426MB@SCN INT8.
• Compact: Based on the CUDA kernels and independent of cutlass.

CUDA BEVFusion

CUDA & TensorRT solution for BEVFusion inference, including:

• Camera Encoder: ResNet50 and finetuned BEV pooling with TensorRT and onnx export solution.
• Lidar Encoder: Tiny Lidar-Backbone inference independent of TensorRT and onnx export solution.
• Feature Fusion: Camera & Lidar feature fuser with TensorRT and onnx export solution.
• Pre/Postprocess: Interval precomputing, lidar voxelization, feature decoder with CUDA kernels.
• Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy.
• PTQ: Quantization solutions for mmdet3d/spconv, Easy to understand.

CUDA CenterPoint

CUDA & TensorRT solution for CenterPoint inference, including:

• Preprocess: Voxelization with CUDA kernel
• Encoder: 3D backbone with NV spconv-scn and onnx export solution.
• Neck & Header: RPN & CenterHead with TensorRT and onnx export solution.
• Postprocess: Decode & NMS with CUDA kernel
• Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy.
• QAT: Quantization solutions for traveller59/spconv, Easy to understand.

CUDA PointPillars

CUDA & TensorRT solution for pointpillars inference, including:

• Preprocess: Voxelization & Feature Extending with CUDA kernel
• Detector: 2.5D backbone with TensorRT and onnx export solution.
• Postprocess: Parse bounding box, class type and direction
• Easy To Use: Preparation, inference, evaluation all in one to reproduce torch Impl accuracy.

CUDA-V2XFusion

Training and inference solutions for V2XFusion.

• Easy To Use: Provides easily reproducible solutions for training, quantization, and ONNX export.
• Quantification friendly：PointPillars based backbone with pre-normalization which can reduce quantization error.
• Feature Fusion: Camera & Lidar feature fuser and onnx export solution.
• PTQ: Quantization solutions for V2XFusion, easy to understand.
• Sparsity: 4:2 structural sparsity support.
• Deepstream sample: Sample inference using CUDA, TensorRT/Triton in NVIDIA DeepStream SDK 7.0.

cuOSD(CUDA On-Screen Display Library)

Draw all elements using a single CUDA kernel.

• Line: Plotting lines by interpolation(Nearest or Linear).
• RotateBox: Supports drawn with different border colors and fill colors.
• Circle: Supports drawn with different border colors and fill colors.
• Rectangle: Supports drawn with different border colors and fill colors.
• Text: Supports stb_truetype and pango-cairo backends, allowing fonts to be read via TTF or using font-family.
• Arrow: Combination of arrows by 3 lines.
• Point: Plotting points by interpolation(Nearest or Linear).
• Clock: Time plotting based on text support

cuPCL(CUDA Point Cloud Library)

Provide several GPU accelerated Point Cloud operations with high accuracy and high performance at the same time: cuICP, cuFilter, cuSegmentation, cuOctree, cuCluster, cuNDT, Voxelization(incoming).

• cuICP: CUDA accelerated iterative corresponding point vertex cloud(point-to-point) registration implementation.
• cuFilter: Support CUDA accelerated features: PassThrough and VoxelGrid.
• cuSegmentation: Support CUDA accelerated features: RandomSampleConsensus with a plane model.
• cuOctree: Support CUDA accelerated features: Approximate Nearest Search and Radius Search.
• cuCluster: Support CUDA accelerated features: Cluster based on the distance among points.
• cuNDT: CUDA accelerated 3D Normal Distribution Transform registration implementation for point cloud data.

YUVToRGB(CUDA Conversion)

YUV to RGB conversion. Combine Resize/Padding/Conversion/Normalization into a single kernel function.

• Most of the time, it can be bit-aligned with OpenCV.
  • It will give an exact result when the scaling factor is a rational number.
  • Better performance is usually achieved when the stride can divide by 4.
• Supported Input Format:
  • NV12BlockLinear
  • NV12PitchLinear
  • YUV422Packed_YUYV
• Supported Interpolation methods:
  • Nearest
  • Bilinear
• Supported Output Data Type:
  • Uint8
  • Float32
  • Float16
• Supported Output Layout:
  • CHW_RGB/BGR
  • HWC_RGB/BGR
  • CHW16/32/4/RGB/BGR for DLA input
• Supported Features:
  • Resize
  • Padding
  • Conversion
  • Normalization

ROI Conversion (ROIs To Continuous Tensor Conversion)

Combine Resize/Padding/Conversion/Normalization into a single kernel function.

• Most of the time, it can be bit-aligned with OpenCV.
  • It will give an exact result when the scaling factor is a rational number.
  • Better performance is usually achieved when the stride can divide by 4.
• Supported Input Format:
  • NV12BlockLinear
  • NV12PitchLinear
  • YUV422Packed_YUYV
• Supported Interpolation methods:
  • Nearest
  • Bilinear
• Supported Output Data Type:
  • Uint8
  • Float32
  • Float16
• Supported Output Layout:
  • CHW_RGB/BGR
  • HWC_RGB/BGR
  • CHW16/32/4/RGB/BGR for DLA input
  • Gray
• Supported Features:
  • Resize
  • Padding
  • Conversion
  • Normalization

Thanks

This project makes use of a number of awesome open source libraries, including:

• stb_image for PNG and JPEG support
• pybind11 for seamless C++ / Python interop
• and others! See the dependencies folder.

Many thanks to the authors of these brilliant projects!