This repository contains MATLAB code for registering a sequence of 3D images by estimating deformation vector fields (DVFs) between an initial volume and subsequent volumes using the pyramidal, iterative Lucas–Kanade optical-flow algorithm.
The animation below shows a lung tumor moving mainly in the superior–inferior direction due to breathing in a 4D computed tomography (4DCT) sequence, along with the time-varying deformation vector fields (DVFs) estimated using this code.
|
|
Left: estimated 3D lung-tumor motion between the reference frame and subsequent frames of a 4DCT sequence, projected onto a coronal plane and displayed over the corresponding 4DCT cross-sections. Right: coronal cross-section from the reference frame of the same sequence, along with the region of interest and tumor contour.
Note: An adaptation of this Lucas–Kanade optical-flow implementation for 2D image registration is available in the
2D-MR-image-predictionrepository, where it is used as the first step of a cine-MR frame-forecasting algorithm.
The input images are located in the Input images folder. This repository includes three chest 4DCT sequences from patients with lung cancer, derived from the TCIA 4D-Lung dataset.
The input image sequences to process are specified in the input_im_dir_suffix_tab array inside the main executable files:
Lucas_Kanade_Pyramidal_Optical_Flow_Main.mLucas_Kanade_Pyramidal_Optical_Flow_Optimization_Main.m
Optical-flow calculation for a specified set of parameters can be performed by executing Lucas_Kanade_Pyramidal_Optical_Flow_Main.m in MATLAB from the repository root.
Program behavior is controlled by manually setting the boolean fields of the beh_par structure defined in load_behavior_parameters3D.m.
Parameters specific to each input image sequence can be found in the following configuration files, located in each input sequence folder:
| Filename | Parameters |
|---|---|
3Dim_seq_par.xlsx |
Image-sequence properties |
3DOF_calc_par.xlsx |
Optical-flow calculation parameters |
3Ddisp_par.xlsx |
Figure display and saving parameters |
The output DVF arrays, visualization files, and performance log files, including the registration root-mean-square error, are saved in the folders Optical flow calculation results mat files, Optical flow projection images, and Log files, respectively. These folders are automatically created if they do not exist.
Note: An optional Python utility, create_gif.py, can be run to assemble selected optical-flow projection images into an animated GIF. The script expects JPG frames named frame_1.jpg, frame_2.jpg, etc. in the Optical flow projection images/out folder and writes output.gif in the repository root.
Grid-search-based hyperparameter selection can be performed by executing Lucas_Kanade_Pyramidal_Optical_Flow_Optimization_Main.m in MATLAB from the repository root.
The hyperparameter grid can be configured in load_3DOF_hyperparameters.m.
Grid-search performance is recorded in DVF optim log file.txt and DVF hyperpar influence [date and time].txt inside the Log files folder.
Running the MATLAB code requires MATLAB's Image Processing Toolbox, as it calls functions such as imgaussfilt3 and dicomread.
The optional GIF-generation utility create_gif.py requires Python with Pillow, imported as PIL.
This repository supports the findings in the following article:
Michel Pohl, Mitsuru Uesaka, Kazuyuki Demachi, Ritu Bhusal Chhatkuli, "Prediction of the motion of chest internal points using a recurrent neural network trained with real-time recurrent learning for latency compensation in lung cancer radiotherapy", Computerized Medical Imaging and Graphics, Volume 91, 2021, 101941, ISSN 0895-6111 [published version] [arXiv]
Two other repositories contain code components supporting the article above:
- Multivariate time-series forecasting with an RNN trained with RTRL: https://github.com/pohl-michel/time-series-forecasting-rtrl
- 3D image warping with Nadaraya–Watson kernel regression: https://github.com/pohl-michel/Nadaraya-Watson-3D-image-warping
Our implementation of the Lucas–Kanade optical-flow algorithm is based on the pyramidal, iterative framework described in the following article:
Jean-Yves Bouguet, "Pyramidal implementation of the affine Lucas Kanade feature tracker description of the algorithm", Intel corporation 5.1-10 (2001): 4 [article]
Please cite the relevant article(s) if you use this code in your research.
This repository is released under the BSD-3-Clause license.