Option --intrinsics-from-views does not work

[jhaggle]

I need to use known camera intrinsics when running Structure From Motion with the command mve.sfmrecon. I therefore add known intrinsics to the meta.ini-files and then run mve.sfmrecon with the option --intrinsics-from-views. But this does not work and mve still finds its own intrinsics and changes the meta.ini-files. I develop this further below.

I start with mve.makescene -i input_dir output_dir. This creates folders with meta.ini-files, one for each image.
The content of the meta.ini-files at this points looks like this:

I then run a script to add my own intrinsics. I also add a variable test just for testing purposes. After running my script the meta.ini-files looks like this:

Next, I run the sfm pointcloud reconstruction with mve.sfmrecon scene_x/ --intrinsics-from-views --fixed-intrinsics. This overwrites the intrinsics that I added to the meta.ini-files, as you can see here:

Thus, the flag --intrinsics-from-views does not work since both focal length and principal point are changed.

[simonfuhrmann]

Sorry for the late response. This is odd. I have to take a closer look this issue if it is still relevant. Unfortunately, I won't get to it in the next couple days. If you manage to resolve this issue, or are able to take a look at this yourself, please let me know.

[jhaggle]

Thank you for your response. I agree that it is odd - as if the argument has no effect. Yes it is absolutely still relevant. I tried to look at the source code but did not manage to locate where the bug is, so it would be very much appreciated if you have to possibility to look at it.

[simonfuhrmann]

Again, sorry for the late response, but I cannot reproduce this. Here is what I did.

1. I took a few photos and ran makescene and sfmrecon on them.
2. I edited view_0000/meta.ini and changed the focal length from some value to precisely 0.81
3. I re-ran sfmrecon --intrinsics-from-views --fixed-intrinsics <scene>

Now I can see in the console:

Adding next view ID 0 (3 of 12)...
Collected 1299 2D-3D correspondences.
Selected 1299 2D-3D correspondences inliers (100%), took 15ms.
Reconstructed camera 0 with focal length 0.81

So it really used the focal length from meta.ini to start the reconstruction. Then after sfmrecon finished, the final meta.ini says something like

$ cat views/view_0000.mve/meta.ini 
# MVE view meta data is stored in INI-file syntax.
# This file is generated, formatting will get lost.

[camera]
focal_length = 0.8100000024
pixel_aspect = 1
principal_point = 0.5 0.5
radial_distortion = 0.05050558224 -0.1205284148
rotation = 0.9566895962 0.1215752214 -0.264508009 -0.1303794533 0.9913363457 -0.01591909491 0.2602810562 0.04971604422 0.9642521143
translation = 1.141109705 -0.002015253529 0.06797743589

[view]
id = 0
name = PXL_20220425_025708260

So indeed, except for some floating point inaccuracies, it did not change the focal length.

Would you mind describing your process in more detail so we can find out what the problem is? Also, maybe try dropping a debug log in Intrinsics::init_from_views in bundler_intrinsics.cc to see if it actually uses that focal length from meta.ini.

[simonfuhrmann]

Also, not to toot my own horn here, but not having used this system in years, the process was pretty smooth. I took 12 photos of a wall in my yard, and the results look quite decent.

[jhaggle]

Hello again Simon and this time I am sorry for the late response. I notice one difference between what you do and what I do.

This is the steps you follow:

1. I took a few photos and ran makescene and sfmrecon on them.
2. I edited view_0000/meta.ini and changed the focal length from some value to precisely 0.81
3. I re-ran sfmrecon --intrinsics-from-views --fixed-intrinsics

Those are the steps that I follow:

1. I run makescene but NOT sfmrecon on my photos
2. I edit all meta.ini-files.
3. I run sfmrecon --intrinsics-from-views --fixed-intrinsics

Thus, I don't run sfmrecon first one time and then a second time.
When running sfmrecon I can see this in the console:

Here is some example output before and after. As you can see, both principal point and focal length are changed.

Before:

After:

[simonfuhrmann]

Ah maybe this is a red herring, but why does your output say PBA:? This string is nowhere to be found in MVE. Are you using the pba branch? It probably doesn't support that.

[jhaggle]

I originally installed MVE using "snap install mve". After your last comment, I was thinking that maybe the snap version of MVE was not updated, and I therefore installed from source. However, the same thing as before still happen, i.e. focal length and principal point change. The output is however a little different. It no longer say "PBA", only "BA":

[simonfuhrmann]

I really only ran sfmrecon before editing meta.ini because I wanted to have a template I can easily modify... but it shouldn't really matter.

Can you provide me with your dataset and all meta.ini files so I can take a look? Maybe just zip your scene and put it somewhere where I can download it?

[jhaggle]

I have now uploaded a folder with images and corresponding metafiles as you asked for.
It can be found here: https://file.io/fH84HxmdWaHT
Or here: https://we.tl/t-lORKsyhG98

The folder contains three copies of the scene-folder. The first after running makescene, the second after also adding intrinsic info (including a test variable “hello”) and the third after running sfmrecon.

The files look as follows (the file view_000.mve is used as example). First after running mve.makescene -i /moving_cube_01/ scene_moving_cube_01 I get this:

Same file after I added some predefined data:

And finally the same file again after running mve.sfmrecon scene_some_images/ --intrinsics-from-views --fixed-intrinsics. As you can see, prinicipal and focal length are changed.

[simonfuhrmann]

From all I can see, both --intrinsics-from-views and --fixed-intrinsics does work with a few caveats that may cause the confusion.

Here is the focal length from your original scene:

$ for name in *.mve; do echo -n $name; echo -n " "; cat $name/meta.ini | grep focal_length; done
view_0000.mve focal_length = 0.8824709947323071
view_0001.mve focal_length = 0.8824709947323071
[...]
view_0014.mve focal_length = 0.8824709947323071
view_0015.mve focal_length = 0.8729114915406176
view_0016.mve focal_length = 0.8729114915406176
view_0017.mve focal_length = 0.8729114915406176

After running sfmrecon, I get the following result:

$ for name in *.mve; do echo -n $name; echo -n " "; cat $name/meta.ini | grep focal_length; done
view_0000.mve focal_length = 0
view_0001.mve focal_length = 0.8824709654
[...]
view_0014.mve focal_length = 0
view_0015.mve focal_length = 0
view_0016.mve focal_length = 0.8729115129
view_0017.mve focal_length = 0.8729115129
[...]

What we see here is that MVE assigns a focal length of 0 to a few views, in particular those views that it couldn't reconstruct. It's not super surprising it didn't reconstruct these views, because they are essentially black frames. Further, the numbers have changed slightly, e.g., from 0.8824709947323071 to 0.8824709654, which is in the range of floating point accuracy. Type float uses 24 bits of precision (excluding sign and exponent), which this is equivalent to only about 7 decimal places, and this is the change we're seeing here. Simply a number load from a text file, storing it in a float, and saving the same number back to file.

I hope this clears things up.

[simonfuhrmann]

Another thing to note is that the SfM feature reconstruction has funky colors, although the input images are gray-scale only. I assume there is a bug somewhere that doesn't handle 1-channel images properly. The final reconstruction looks somewhat reasonable too.

[jhaggle]

Thank you for your quick response. Even if the differences in focal lengths are in the range of floating point accuracy, that does not apply to the principal points.

Here are some of the initial principal points from my original scene:

view_0000.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0001.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0002.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0003.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0004.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0005.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0006.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0007.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0008.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0009.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0010.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0011.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0012.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0013.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0014.mve principal_point = 0.4967022112105042 0.5037006649584649
view_0015.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0016.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0017.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0018.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0019.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0020.mve principal_point = 0.49673967040143907 0.5087240861539177
view_0021.mve principal_point = 0.49673967040143907 0.5087240861539177

After running sfmrecon, principal points are always changed to 0.5 0.5:

view_0000.mve principal_point = 0.5 0.5
view_0001.mve principal_point = 0.5 0.5
view_0002.mve principal_point = 0.5 0.5
view_0003.mve principal_point = 0.5 0.5
view_0004.mve principal_point = 0.5 0.5
view_0005.mve principal_point = 0.5 0.5
view_0006.mve principal_point = 0.5 0.5
view_0007.mve principal_point = 0.5 0.5
view_0008.mve principal_point = 0.5 0.5
view_0009.mve principal_point = 0.5 0.5
view_0010.mve principal_point = 0.5 0.5
view_0011.mve principal_point = 0.5 0.5
view_0012.mve principal_point = 0.5 0.5
view_0013.mve principal_point = 0.5 0.5
view_0014.mve principal_point = 0.5 0.5
view_0015.mve principal_point = 0.5 0.5

As regards the funny colors I will try to save the images as color images instead.

[simonfuhrmann]

That is a good point you make. This is something that may actually not be supported. While our bundle adjustment can certainly not optimize the principal point, I don't see a reason why it can't use the principal point from the intrinsics in meta.ini, so I think that could be implemented. I am not sure when I find the time and patience to try this out, but I'll make a mental note for this.

[simonfuhrmann]

@flanggut I believe in bundle_adjustment.cc itself we don't even use the principal point at all, because we already assume the camera is centered in the coordinate system appropriately, and the principal axis is going straight down the z axis. Do you have any idea how we could incorporate the principal point then? Do we have to provide a way to specify an additive shift on the image plane after projection?