Problem using camera matrices to project camera points to world

[rhodriguerrier]

I need to download the intrinsics and extrinsics matrices when generating the point tracking kubric dataset (defined in challenges/point_tracking/dataset.py). I simply return the variables intrinsics and matrix_world. However, when I use these to try and project the camera coordinates to world coordinates, they do not seem to be correct (e.g. static points move in the world when they should not and points on the floor curve rather than sitting on a flat plane). The below code shows how I have been trying to compute the world coordinates for each camera pixel using the depth_map that I returned from the dataset.py file. Am I doing something wrong? No matter whether I invert the extrinsics or not, all the points seem to be flattened into a plane. Thanks

# Copied from challenges/point_tracking/dataset.py
depth_map = data["depth_map"].reshape(24,256,256)
depth_range = data["depth_range"]
depth_range_f32 = depth_range.astype(np.float32)
depth_min = depth_range_f32[0]
depth_max = depth_range_f32[1]
depth_f32 = depth_map.astype(np.float32)
depth_map_new = depth_min + depth_f32 * (depth_max-depth_min) / 65535

fx = data['intrinsics'][frame_idx][0, 0]
fy = data['intrinsics'][frame_idx][1, 1]
x0 = data['intrinsics'][frame_idx][0, 2]
y0 = data['intrinsics'][frame_idx][1, 2]
extrinsics = data["extrinsics"][frame_idx]

for y in range(0, depth_map_new.shape[1], 5):
    for x in range(0, depth_map_new.shape[2], 5):
        z = depth_map_new[frame_idx, y, x]

        x_cam = (x - x0) * z / fx
        y_cam = (y - y0) * z / fy

        camera_coord = np.array([[x_cam, y_cam, z, 1.0]])
        world_coord = np.matmul(np.linalg.inv(extrinsics), np.transpose(camera_coord, (1, 0)))