cube map renderer gets stuck in have_invalid_color_pixels while loop

[raphaelsulzer]

Hi, and thanks for the great tools you provided!

I'm trying to apply the whole pipeline on the Tanks and Temples dataset to evaluate with your metrics.

However, I get stuck in the first step when trying to render the cube map images. The code doesn't pass the following while loop:

dataset-pipeline/src/exe/cube_map_renderer.cc

Line 328 in a2223be

while (have_invalid_color_pixels) {

I would appreciate any inside what to do.

Kind regards!

[ClementPinard]

Hi, what data are you using to run the cube map render the ply ground truth that can be downloaded here in section training data ? Because even though it accepts any PLY file, this program expects raw Faro focus scans converted in ply, and anything else might end up with un expected outcome.

More particularly, it expects 3D points to be all around the coordinate (0,0,0) and there will always be projected points for each face of the cube rendered. I suspect that in this case, there was a face with only invalid pixels.

What you can do to solve this is to check that the validity map is not full of False booleans before entering in this while loop.

Another thing you can try might be a bit more complicated, but would reflect more the original usecase of ETH3D. As tanks and temples was also constructed with Faro focus scans but we can only access them already stitched together, you can to try to localize from where each faro focus scan was performed (there is a circular pattern on a floor) and deduce the coordinate of the center of projection, and shift the whole point cloud so that it is now at (0,0,0). Of course this can be done inderictly by changing the equation used here to do a simple substraction.

In the end you would get a program that would take a 3D coordinate for argument in addtiion to the already existing ones for cubeMapRender.

[raphaelsulzer]

Thank you for the quick reply!

Hi, what data are you using to run the cube map render the ply ground truth that can be downloaded here in section training data ?

In fact I am using the individual scans of the dataset which are available on the bottom of the page you linked. They also include the sensor position.

Of course this can be done inderictly by changing the equation used here to do a simple substraction.

I did this:
Eigen::Vector3f r_p = R * (point.getVector3fMap()-sensor_center.getVector3fMap());

However, the filled_in_validity_map is indeed full of False booleans before entering the while loop, even after applying this fix.

I would have guessed that the issue is simply that the scans are cleaned in a way that there are only points in one direction, thus most of the cube maps are empty. But I am not getting a single valid image.

Any other ideas?

[ClementPinard]

Ha I didn't see the individual scans ! I'll try to test them ASAP (in one week). In the mean tie, maybe you can try the terrace scan and spot the difference between the two sets of PLY files ? Hopefully there was no regression on CubeMapRenderer and it works as intended, but i apply some heavy changes lately, and even though it did not reach the CubeMapRenderer code, there was no test to check regression on this program, and i did not test it afterwards.

Thanks for your help !

[raphaelsulzer]

Thank you!

In the mean tie, maybe you can try the terrace scan and spot the difference between the two sets of PLY files ?

I did that, and cannot find any differences that justify the wrong behaviour. Rendering the cube maps for the terrace scan works fine.

Here you can find my code for loading the individual scans together with their sensor position from the sensor_pos.txt file that you can download from the Tanks and Temples website. Maybe it helps.

[ClementPinard]

Hi, thanks for your implementation. Just tested it, and the problem is that all points are read as (0,0,0). Which is obviously a problem. Turns out the problem is that points are saved as float64, a format PCL is not compatible with, it expects float32 numbers. Here is a discussion about it : PointCloudLibrary/pcl#1953

The obvious workaround is to convert them to float32 PLY files, e.g. with CloudCompare :

CloudCompare -SILENT -O Barn01.ply -C_EXPORT_FMT PLY -SAVE_CLOUDS

(see here https://www.cloudcompare.org/doc/wiki/index.php?title=Command_line_mode )

Now I tested it, and we get some funky results (see attached), with the 3d model being far from filling the field of view and thus creating heavy artefacts. Some more tests may be needed.

Clément

[raphaelsulzer]

Hi Clément,

just to let you know, I have been too busy in the last two weeks to take another look at this, but I do still want to get this to work. I'll investigate further soon and keep you posted.

Thanks for your help so far,
Raphael