Dense image inverse warping #156
Description
Hi,
I am having difficulty with defining my Energy.
Here is the problem: I would like to perform image warping via known depth map and camera pose. The algorithm should look like this:
## Backproject points
# Create mesh grids (D: depth map; inv_cam_mat: inverse camera intrinsics)
x_grid <- [[0, 1, ..., w], ...., [0, 1, ..., w]] * D
y_grid <- [[0, 0, ...., 0], ..., [h, h, ..., h]] * D
ones <- [[1, 1, ...., 1], ..., [1, 1, ..., 1]] * D
pixel_coords <- [[x_grid_flatten], [y_grid_flatten], [ones]]
local_pointcloud <- inv_cam_mat X pixel_coords # shape: 3, w*h
## Re-project points
# Create projection matrix
proj <- cam_mat X cam_pose # shape: 3, 4
rot <- proj[:, :3] # shape: 3, 3
trans <- proj[:, 3] # shape: 3, 1
trans_broadcasted <- trans with duplicated copies along 2nd dims # shape: 3, w*h
# Get pixel locations on the source image
pixel_coords_warped <- rot X local_pointcloud + trans_broadcasted
pixel_coords_warped_norm <- pixel_coords_warped / pixel_coords_warped[2, :]
# Inverse warped image
I_hat = I_source[pixel_coords_warped_norm[0], pixel_coords_warped_norm[1]]
# Pytorch implementation can be found here: https://github.com/ClementPinard/SfmLearner-Pytorch/blob/master/inverse_warp.py
To convert the algorithm to the Terra/Lua code, I need the matrix multiplication and matrix slicing techniques, assuming I can create mesh grid and matrix broadcasting in my C++ code. I would also need bilinear interpolation to get the pixel intensities at sub-pixel locations (I presume that the optical flow example is already doing that, how can I use the code without the autodiff stuff?). I am a total beginner in Lua/Terra language, I hope that you can point me to the right resources.
Alternatively, can I just create an API to perform inverse_warp using Pytorch and spit out the inverse warped image for energy computation? If it is possible, how should I create the pipeline so that the datatypes match in both ends (i.e., the .t file and the .py file)?