Camera 2D LiDAR Calibration Library

This is a camera and 2D LiDAR calibration library that computes the camera's intrinsic matrix and distortion coefficients as well as the SE(2) transformation from 2D LiDAR frame to the camera frame.

A set of example ROS bags are made available in this Google Drive folder. We also provide a useful tool that extracts images and point clouds in this package. This package will be used regularly in these calibratoin processes to obtain the sensor measurements.

Camera Intrinsic Calibration

The camera intrinsic calibration script is here. It reads in all the pictures in a user-designated folder and conducts camera intrinsic calibration, expecting all the input images to contain a fully visible checkerboard with a known size and dimension.

Dependencies

This camera intrinsic calibration script relies on very few packages, and all of them should have already been installed with python and ROS2 (assuming full installation). However, since this is not a ROS dependent package, we still explain the following required dependencies:

• OpenCV - You should not need to do this as OpenCV should have already been installed via the cv_bridge package within ROS.
• numpy

Experiment Procedure

This camera intrinsic calibration method follows the procedure detailed below:

1. A square checkerboard pattern is within full-view of the camera, and supported by a wall (a hard surface). A copy of this checkerboard is provided in checkerboard.pdf. You need to know the dimensions of this checkerboard, specifically the checkerboard square's size and the layout of the squares. You will need to adjust the script cam_intrinsic.py to reflect that.
2. Record a ROS bag where you hold the checkerboard in the view of the camera and move the board around. You should pause at different poses to make sure the camera capture a clear image of the checkerboard, and make sure that the whole board is within the camera's field of view. An example bag is provided here.
3. Use the ROS bag image extraction tool to obtain all the images in the bag, and select at minimum 10 images out of them. Put them in a separate folder.
4. Run the camera calibration script with the image_dir input parameter pointing at the folder where you placed your selected images. Make sure you use the correct checkerboard properties within the script.
5. Obtain the camera extrinsics from the print out.

Running the script

Run at the top directory level of this package folder:

python camera_2d_lidar_calibration/cam_intrinsic.py <path/to/image/folder>

TODOs

Within the cam_intrinsic.py script, you are expected to update the properties of the checkerboard you use in your calibration process, i.e. use the width and height of checkerboard vertices and size of the blocks that matches with the actual board.

Camera 2D LiDAR Calibration

The camera LiDAR extrinsic calibration script is here. It reads in all the images and point clouds from two user-designated folders to calibrate for the relative 2D/SE(2) transformation between the camera and the 2D LiDAR using ICP. It expects all the input images to contain a fully visible checkerboard with a known size and dimension, as well as the majority of the checkerboard wall in the camera view.

Installation and Dependencies

To install, run the following code:

mkdir git
cd git
git clone https://github.com/ACFR-RPG/camera_2d_lidar_calibration.git
cd camera_2d_lidar_calibration
pip install -e . # install python dependencies

This script depends on the following packages:

• OpenCV - You should not need to do this as OpenCV should have already been installed via the cv_bridge package within ROS.
• Open3D
• numpy
• scipy
• tkinter - built into python
• scikit-learn
• matplotlib

Running

Run at the top directory level of this package folder:

python camera_2d_lidar_calibration/cam_lidar_2d_icp.py <path/to/image/folder> <path/to/cloud/folder>

Assumptions and Environment Setup

This camera LiDAR extrinsic calibration method relies on the following environment setup and assumptions:

1. A square checkerboard pattern is within full-view of the camera, and supported by a wall with straight edges. A copy of this checkerboard is provided in checkerboard.pdf. The software does not necessarily require the dimensions of the wall, but it needs dimensions of the checkerboard; specifically the checkerboard square's sizes and the total number of squares.
2. Corners on the checkerboard will be used to determine the pose of the checkerboard, and in turn estimate a horizontal line that can represent the wall. Any line defined by a row of corners should be parallel with the ground plane and orthogonal to the edges of the wall.
3. The checkerboard is orthogonal to the ground plane.
4. The LiDAR is scanning the world in a plane that is parallel to the ground. The LiDAR can be placed at any height, so long as its rays intersect with the wall.
5. The left and right edges of the wall should be visible in the calibration image, and are discernible from the surroundings. However this is not a hard requirement, but a recommendation.

To see a typical example that satisfies the assumptions above, refer to the following image corresponding to TurtleBot3 Burger. Its 2D LiDAR is located at the top of the robot and scans the world in a plane parallel to the ground.

In this example, both the checkerboard and the left/right edges of the wall are within full-view.

Camera POV (left) and RViz2 Visualisation of LiDAR Scans (right). In the LiDAR scans, the bottom three sharp lines correspond to the boxes, and the top line corresponds to the wall.

Calibration Routine

After setting up the environment, the following routine will produce the required ROS bags that will be fed into the library.

1. Obtain the intrinsic and distortion parameters of the camera through any standard camera calibration method, e.g. the camera intrinsic calibration process above. Ensure the units of these parameters agree with the units of the checkerboard. We recommend using meter (m) as the standard unit.
2. Setup the camera in an environment where you can recognise some features or structure in the LiDAR data (e.g. the boxes surrounding the TurtleBot in the picture above). This setup will assist in the calibration process later.
3. Start recording a ROS bag that listens to both the camera image topic /camera/image_raw and LiDAR topic /pointcloud2d.
4. Set up the checkerboard wall, making sure the whole checkerboard is visible in the camera view and the wall is vertical to the ground plane. Keep it stationary for a few seconds.
5. Move the wall and checkerboard to a different position, making sure the whole checkerboard is visible in the camera view and the wall is vertical to the ground plane. Keep it stationary for a few seconds.
6. Repeat step 5 about 5-10 times.
7. Stop recording the ROS bag, and extract all the images and point clouds from the bag using scripts provided in this package. For each wall position collected in the dataset, select a pair of corresponding camera view and the LiDAR scan (matched using timestamps, for example), and put the images and point clouds in two separate folders.
8. Run the camera LiDAR calibration script, specifying the image_dir and laser_dir arguments, which correspond to <path/to/image/folder> and <path/to/cloud/folder> respectively. Make sure the camera intrinsic and distortion parameters here and checkerboard parameters here agree with your experiment setup.
9. Use the interactive GUI to confirm the checkerboard detection and select corresponding LiDAR points for each image pair.
10. Obtain the calibration results in program print out.

Alternatively, you can also record a separate ROS bag for 10-20 seconds for each checkerboard pose, like what we did in the example dataset here. You will need to run the image and point cloud extraction script for each bag in this case, and each bag will yield you a pair of corresponding camera view and point cloud.

TODOs

Within the cam_lidar_2d_icp.py script, you are expected to update the properties of the checkerboard you use in your calibration process here, i.e. use the width and height of checkerboard vertices and size of the blocks that matches with the actual board, as well as use the correct camera intrinsic by adjusting the parameters here.

We further provide an initial transformation to assist ICP, if needed. The transformation can be edited here. This is not mandatory. You are welcome to keep the initial_tf as identity if your ICP result is good.

Interactive Interface

2026_mtrx5700_assignment2_cameralidarcalibration.mp4

The video above shows what you should expect to see once you are running the script. This example uses the images and point clouds in the /example folder. The script will read in images and point clouds and first try to extract a checkerboard from the image. If it can successfully extract the checkerboard, it will launch an interface, asking you to confirm that it is a good checkerboard. You are recommended to zoom in and check whether the vertices of the checkerboard are correctly detected. If not, you may want to check whether your checkerboard corner detection/searching line is implemented correctly. If it is wrong, clicking the This looks wrong! button will terminate the interface.

If the checkerboard is correctly detected, clicking Done will notify the script to extract and estimate a line in 3D space that correspond to the y axis of the checkerboard pose. Then, it will launch an interface asking you to select the matching point cloud. You should use the Zoom function again to only include the points corresponding to the wall in the interface, and click Select Points to highlight them. Then you can click Done to record them into the system.

Repeat this process for all pairs of images and LiDAR scans. At the end, the script will provide visualisation for introspection and qualititative evaluation. First, we will visualise the points/lines that represent the checkerboard poses in both camera view (estimated) and LiDAR scan (selected). Then, we will visualise the 2D ICP results by aligning the point clouds using computed transformations from two different ICP algorithms.