Question about camera matrix

[liuxiao916]

Dear authors,

Thank you for your excellent work. I have a question regarding the accuracy of the camera_matrix used for transforming depth maps into world-space point clouds.

I noticed that the camera matrix is obtained using the following line:

camera_matrix = self.agents[agent_idx].dynamic.camera_matrix.reshape((4, 4))

Then we can get the point cloud like this

TDWUtils.get_point_cloud(depth=depth_values, camera_matrix=camera_matrix)

Expected Behavior:
Given that the camera matrix represents the pose of the camera in the world, I would expect that all generated point clouds should be aligned in the same world coordinate system.

Actual Behavior:
In practice, the point clouds generated for each frame are not aligned — they appear to have angular deviations relative to each other, as shown in the attached visualization.

Question:
Could there be an issue with the definition or interpretation of the camera_matrix? Do you have any suggestions or guidelines on how to correctly convert points from the camera frame to the world frame?

Any clarification or recommended fix would be greatly appreciated.

Thank you!

[alters-mit]

Wow I've never seen that before... Do you see the same thing when you run the example controller? https://github.com/threedworld-mit/tdw/blob/master/Python/example_controllers/visual_perception/depth.py