Performance Evaluation of 3D Object Detection Models in a Simulation-Based Generated Dataset of a SOTIF-Related Use Case
This repository contains the LiDAR dataset and scripts used for evaluating State-of-the-Art (SOTA) 3D object detection models in a simulation-based generated dataset for a SOTIF-related use case. The dataset follows the KITTI format and is generated using CARLA, simulating 21 diverse weather conditions across different times of the day.
The evaluation is conducted using pre-trained models on KITTI, tested with MMDetection3D and OpenPCDet toolkits to assess performance variations under simulated conditions.
This repository is organized as follows:
- ImageSets/ - training/testing splits
- kitti_gt_database/ - ground truth database
- testing/ - testing dataset
- kitti_dbinfos_train.pkl - training dataset information
- kitti_infos_test.pkl - test dataset info
- kitti_infos_train.pkl - train dataset info
- kitti_infos_trainval.pkl - train+validation dataset info
- kitti_infos_val.pkl - validation dataset info
- carla_data_descriptor.py - describes dataset properties
- carla_weather_presets.txt - weather settings for simulation
- CARTI_Dataset_V1.0.py - dataset generation script
- CMM_CARLA_Config.py - CARLA configuration script
- lane_change.py - lane change maneuver script
- LaneChange.xml - scenario configuration
- README.md - project documentation
- requirements.txt - dependencies
- LICENSE - MIT license
- dataset_snapshot.png - dataset preview
- scenario_snapshot.png - simulation environment snapshot
- OpenPCDet/
- mmdetection3d/
This project simulates a SOTIF-related use case in a multi-lane highway scenario where an Ego-Vehicle (LiDAR-equipped) navigates under varying weather conditions. The key elements of the scenario include:
- Ego-Vehicle (Blue) drives in Lane 3, adjusting speed based on traffic.
- Fast-moving vehicle (Red, 90 km/h) overtakes a slow-moving vehicle (Green, 60 km/h).
- Ego-Vehicle detects the slow vehicle and decelerates to avoid collision.
- LiDAR sensor performance is evaluated under 21 different weather conditions.
- Dataset records LiDAR point cloud frames in KITTI format, ensuring compatibility with benchmark models.
📸 Scenario Visualization:
Here’s a 2D visualization of a sample LiDAR point cloud frame from the dataset:
📸 Dataset Sample:
- 547 frames of LiDAR point cloud data formatted in KITTI standard.
- Simulated using CARLA, with custom weather conditions.
- Dataset split into training, validation, and test sets.
- Pre-trained KITTI-based models applied using:
- MMDetection3D
- OpenPCDet
- Average Precision (AP)
- Recall
- Intersection over Union (IoU) Thresholds (0.30, 0.50, 0.70)
- KITTI-trained models show performance variations under different simulated weather conditions.
- Domain gap between real-world KITTI data and simulation-based data is analyzed.
- Detailed results are presented in the research papers cited below.
This dataset and methodology were used in the following research papers:
📖 Patel, Milin and Jung, Rolf
"Simulation-Based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Case"
🚀 Proceedings of the 10th International Conference on Vehicle Technology and Intelligent Transport Systems (VEHITS 2024)
🔗 DOI: 10.5220/0012707300003702
📖 Patel, Milin and Jung, Rolf
"Uncertainty Representation in a SOTIF-Related Use Case with Dempster-Shafer Theory for LiDAR Sensor-Based Object Detection"
🚀 arXiv preprint, 2025.
🔗 arXiv Link
🔗 License This project is licensed under the MIT License. See LICENSE for details.
📬 Contact For any inquiries or collaborations, please reach out via milinp101996@gmail.com or open an issue on GitHub.