Something confusing when computing the overlapping mask

[ChenYutongTHU]

Hi authors,

Thanks for this interesting work!

I have a question regarding the code for the overlapping mask computation.

Here, you compare the predicted depth map of view j, i.e. curr_depth_map, with the depth maps of previous views, but they are in different camera coordinate systems.

InstantSplat/utils/sfm_utils.py

Lines 347 to 355 in 3c9454a

	points_2d, _ = project_points(points, camera_intrinsics, extrinsics_w2c)
	# Check if points are within image bounds
	valid_points = (points_2d[:, 0] >= 0) & (points_2d[:, 0] < w) & \
	(points_2d[:, 1] >= 0) & (points_2d[:, 1] < h)
	# Check depth consistency using vectorized operations
	valid_points_2d = points_2d[valid_points].astype(int)
	valid_depths = depths[valid_points]

InstantSplat/utils/sfm_utils.py

Lines 361 to 364 in 3c9454a

	depth_differences = np.abs(valid_depths - curr_depth_map[y_coords, x_coords])
	# Create a mask for points where the depth difference is below the threshold
	consistent_depth_mask = depth_differences < depth_threshold

According to the paper, should we compute the difference between curr_depth_map and the projected depth map, e.g. the ignored 2nd returned item '_' here?

InstantSplat/utils/sfm_utils.py

Lines 347 to 348 in 3c9454a

	points_2d, _ = project_points(points, camera_intrinsics, extrinsics_w2c)

Please let me know if I miss something here.

Great thanks!