The code provided in this repository was used to generate motion blur datasets for the paper "On Motion Blur and Deblurring in Visual Place Recognition" (see citation below). However, this code base can be used on any GoPro videos to generate motion blur datasets and benchmarking data to test VPR algorithms.
The main entry point of this code base is the script main.py.
main.py takes several commands: gps, extract, avg-blur, prune, loops.
The following sections are intended to be a guide to get benchmarking dataset from raw GoPro videos.
At the bottom of this README file you will find the link to the dataset used for the paper "On Motion Blur and Deblurring in Visual Place Recognition".
Please note that this function incorporates the code from pygmf.
Important Note: Extracting GPS data from GoPro videos works only if they lack an audio track. If your video has an audio track, you must remove it prior to running the GPS extraction step.
Usage:
python main.py gps -v /home/main/Documents/GX010091.MP4 -o /home/main/Documents/GX010091_GAPS.CSV
The resolution of GPS might be lower than FPS. This happens, for example, for 240FPS videos. To fill the gaps between frames add the flag --fill-the-gaps:
python main.py gps -v /home/main/Documents/GX010091.MP4 -o /home/main/Documents/GX010091_GAPS.CSV --fill-the-gaps
Below it is an example of CSV with filled gaps. The frames 0 and 13 are real GPS data (look at the true_gps column). The frames in the between are reported with GPS data replicated. This is done to have a complete list of the frames to facilitate the ground truth creation.
idx,seq,true_gps,frameId,label,latitude,longitude,elevation,speed,pos_diluition,datetime,yy,MM,dd,hh,mm,ss,us
0,S0,1,0,frame_000000,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.755000,2023,10,3,12,53,46,755000
1,S0,0,1,frame_000001,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.759170,2023,10,3,12,53,46,759170
2,S0,0,2,frame_000002,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.763340,2023,10,3,12,53,46,763340
3,S0,0,3,frame_000003,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.767510,2023,10,3,12,53,46,767510
4,S0,0,4,frame_000004,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.771680,2023,10,3,12,53,46,771680
5,S0,0,5,frame_000005,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.775850,2023,10,3,12,53,46,775850
6,S0,0,6,frame_000006,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.780020,2023,10,3,12,53,46,780020
7,S0,0,7,frame_000007,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.784190,2023,10,3,12,53,46,784190
8,S0,0,8,frame_000008,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.788360,2023,10,3,12,53,46,788360
9,S0,0,9,frame_000009,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.792530,2023,10,3,12,53,46,792530
10,S0,0,10,frame_000010,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.796700,2023,10,3,12,53,46,796700
11,S0,0,11,frame_000011,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.800870,2023,10,3,12,53,46,800870
12,S0,0,12,frame_000012,44.9420131,10.6758726,19.462,17.93,1.35,2023-10-03 12:53:46.805040,2023,10,3,12,53,46,805040
13,S0,1,13,frame_000013,44.942005,10.6758678,19.463,17.99,1.35,2023-10-03 12:53:46.810556,2023,10,3,12,53,46,810556
14,S0,0,14,frame_000014,44.942005,10.6758678,19.463,17.99,1.35,2023-10-03 12:53:46.814726,2023,10,3,12,53,46,814726
15,S0,0,15,frame_000015,44.942005,10.6758678,19.463,17.99,1.35,2023-10-03 12:53:46.818896,2023,10,3,12,53,46,818896
A typical usage is as follow:
python main.py extract -v YOUR_VIDEO_FULLPATH -o DESTINATION_DIR -m MUTE_VIDEO_FULLPATH -W TARGET_WIDTH -H TARGET_HEIGHT -f IMAGE_FORMAT
- YOUR_VIDEO_FULLPATH: it is the go-pro video to unfold into the frames
- DESTINATION_DIR: the folder where all the frames will be stored
- MUTE_VIDEO_FULLPATH: the actual extraction of the frames is from the video without the audio. This file specifies the destination of such a muted video. It is not required.
- TARGET_WIDTH: reshapes the extracted frame. It is not required.
- TARGET_HEIGHT: reshapes the extracted frame. It is not required.
- IMAGE_FORMAT: jpg, png, etc...
Some examples:
The setting used to produde the dataset for the paper.
python main.py extract -v go_pro/video_01.mp4 -o go_pro/video_01_frames -m video_01_no_audio.mp4 -W 960 -H 540 -f jpg
If you do not need the no-audio video:
python main.py extract -v go_pro/video_01.mp4 -o go_pro/video_01_frames --clean-mute-video -W 960 -H 540 -f jpg
If you do not need any reshape:
python main.py extract -v go_pro/video_01.mp4 -o go_pro/video_01_frames --clean-mute-video -f jpg
More details can be found in the parse_args() of main.py
python main.py avg-blur -d FOLDER_WITH_THE_FRAMES -o OUT_FOLDER -b BLUR_INTENSITIES
- FOLDER_WITH_THE_FRAMES: it is the directory where the source frames are stored. It corresponds to DESTINATION_DIR of the above section.
- OUT_FOLDER: is where the blurred frames are stored
- BLUR_INTENSITIES: a list of integers that specify the number of frames to average. The higher, the more intense the blur effect. For example, 040 produces a blurred frame by averaging 40 consecutive frames in FOLDER_WITH_THE_FRAMES
Here is an example:
python main.py avg-blur -d go_pro/video_01_frames -o go_pro/video_01_blurred_frames -b 040 080 120 240
This command produces four blurred versions of the video frames into go_pro/video_01_blurred_frames
If you need to reduce the number of frames to use in one of your experiment, you can delete some of them. It should be used only on sharp frames from an high FPS (For example on go_pro/video_01_frames) to produce slimmer dataset for some experiments (e.g. reference loop in VPR).
python main.py prune -d FOLDER_WITH_THE_FRAMES -o LOW_FPS_FOLDER --fps=TARGET_FPS --sfps=FPR_OF_THE_SOURCE --override
- FOLDER_WITH_THE_FRAMES: it is the directory where the source frames are stored. It corresponds to DESTINATION_DIR of one of the above section.
- LOW_FPS_FOLDER: is where the pruned dataset is stored
- TARGET_FPS: the target fps of the pruned dataset
- FPR_OF_THE_SOURCE: how many frames in FOLDER_WITH_THE_FRAMES for a second of video
Here is an example:
python main.py prune -d go_pro/video_01_frames -o go_pro/pruned_video_01 --fps=1 --sfps=240 --override
The video where recorded and 240 FPS. We want to keep only 1 frame per second, that is 1 out 240.
This section describes the end-to-end process for generating motion blur benchmark datasets from your raw videos. It relies on executing the commands documented above in a specific order. You can automate these steps by referring to the scripts provided in the scripts folder (e.g., generate_blur.bat and assemble_benchmark.bat).
Note
To facilitate these operations, we provide .bat files (which are fully tested) and .sh files (which are currently untested and provided as-is) in the scripts/ directory.
look for:
- scripts/generate_blur.bat (or sh if you are on linux)
- scripts/assemble_benchmark.bat (or sh if you are on linux)
Create a root directory for your dataset (e.g., <your_path>\GUASTALLA-03) and set up the following subdirectories:
raw_videos: Place your source.mp4videos here.mute_videos: Destination for videos with the audio track removed.gps: Destination for the extracted GPS data (.txtor.csv).unrolled: Destination for the extracted original frames.blurred: Destination for the frames with the averaging blur effect applied.pruned_blurred: Destination for the pruned blurred frames.low_fps: Destination for the sharp low-fps reference frames.benchmark: The final destination where the assembled loop pairs (query and reference) will reside.
For the examples below, let's assume:
- Dataset Root:
<your_path>\GUASTALLA-03 - Dataset Name / Prefix:
GUASTALLA-03 - Sequence/Video Name:
GUASTALLA-03-05 - Source Video:
<your_path>\GUASTALLA-03\raw_videos\video_05.mp4
Extract the GPS track from the original video. Crucial requirement: GoPro GPS extraction works only if the video lacks an audio track. Either provide a video file that has already been muted, or ensure your source video has no audio.
python main.py gps -v <your_path>\GUASTALLA-03\raw_videos\video_05.mp4 -o <your_path>\GUASTALLA-03\gps\GUASTALLA-03-05.txt --fill-the-gaps
Unroll the video into its constituent frames. This will also output a muted video which can be useful.
python main.py extract -v <your_path>\GUASTALLA-03\raw_videos\video_05.mp4 -o <your_path>\GUASTALLA-03\unrolled\GUASTALLA-03-05 -m <your_path>\GUASTALLA-03\mute_videos\MUTE_video_05.mp4 -W 960 -H 540 -f jpg
Create different blur intensities by averaging multiple consecutive frames. In this example, we generate intensities for 10, 20, 30, 40, 60, 80, 120, and 240 frames.
python main.py avg-blur -d <your_path>\GUASTALLA-03\unrolled\GUASTALLA-03-05 -o <your_path>\GUASTALLA-03\blurred\GUASTALLA-03-05 -b 10 20 30 40 60 80 120 240
(You must repeat this for every blur level, or pass them as a list depending on your platform/script automation)
The blurred sequences must be pruned to simulate a specific FPS. Here we assume 240 FPS source and we prune to match the blur length itself or simply down to 1 FPS for benchmark generation. The scripts do this per blur level (<bl>):
python main.py prune -d <your_path>\GUASTALLA-03\blurred\GUASTALLA-03-05\AVGBLUR_B-<bl> -o <your_path>\GUASTALLA-03\pruned_blurred\GUASTALLA-03-05\<bl> --fps=<bl> --sfps=240 --override
(Replace <bl> with the respective blur intensity like 010, 020, etc.)
Extract a set of sharp reference frames at 1 FPS from the unrolled sequence to act as the ground truth.
python main.py prune -d <your_path>\GUASTALLA-03\unrolled\GUASTALLA-03-05 -o <your_path>\GUASTALLA-03\low_fps\GUASTALLA-03-05 --fps=1 --sfps=240 --override
Once you have generated at least one Query sequence (e.g. GUASTALLA-03-05) and one Reference sequence (e.g. GUASTALLA-03-03) through the steps above, you can pair them to assemble the final benchmark.
First, copy the sharp 1 FPS frames into a 001 subfolder of your query, to act as the baseline unblurred sequence:
mkdir <your_path>\GUASTALLA-03\pruned_blurred\GUASTALLA-03-05\001
xcopy /E <your_path>\GUASTALLA-03\low_fps\GUASTALLA-03-05 <your_path>\GUASTALLA-03\pruned_blurred\GUASTALLA-03-05\001
Then run the loops command to match the query against the reference using their GPS traces:
python main.py loops -R <your_path>\GUASTALLA-03\low_fps\GUASTALLA-03-03 -o <your_path>\GUASTALLA-03\benchmark\GUASTALLA-03-05_to_03 -B <your_path>\GUASTALLA-03\pruned_blurred\GUASTALLA-03-05 --ref_gps="<your_path>\GUASTALLA-03\gps\GUASTALLA-03-03.txt" --query_gps="<your_path>\GUASTALLA-03\gps\GUASTALLA-03-05.txt" --blur-prefix="" -b 1 10 20 30 40 60 80 120 240
This final command creates the evaluation-ready dataset pairing the queries and references together.
You can download the datasets used in the paper from here.
The dataset consists of three locations captured at several blur intensities. For each localtions the are two distinct pairs of query and reference loops as described in the paper.
You can use the code of this repository to generate any number of blur levels starting from your own GoPro videos (240 fp are reccomanded) to generate your own blurred dataset.
This code was used to produce the paper "On Motion Blur and Deblurring in Visual Place Recognition". If you use this code or the resulting datasets in your research, please cite our work:
@ARTICLE{10938388,
author={Ismagilov, Timur and Ferrarini, Bruno and Milford, Michael and Tan Viet Tuyen, Nguyen and Ramchurn, Sarvapali D. and Ehsan, Shoaib},
journal={IEEE Robotics and Automation Letters},
title={On Motion Blur and Deblurring in Visual Place Recognition},
year={2025},
volume={10},
number={5},
pages={4746-4753},
keywords={Visual place recognition;Videos;Robots;Benchmark testing;Meteorology;Lighting;Sensors;Visualization;Trajectory;Training;Localization;vision-based navigation;data sets for robotic vision},
doi={10.1109/LRA.2025.3554103}
}