Reconstruct.py & Demo.ipynb are in conflict with each other and broken // Built the project but couldn't verify results

[PoyrazTahan]

Hey Authors, First of all I would like to start by, I loved your approach in terms of using Colmap and combination of newer depth tecqniques to first get a very close camera position and then putting the depths and finalize with regularizing the on normals. Brilliant approach. Especially I am very impressed with your first gif in READ.md.

But after 76 hours of tinkering I am giving up, because I was only able to recreate the demo.ipynb. Which seems to be working for 6 images fairly well but rest of the benchmark and anything my custom dataset fails and I am not sure if I am doing something fundamentally wrong or the pipeline is broken.

First (Input):

• Beside from [images and intrinsics] of the camera, did you use anything to be able to produce the example outputs?
  Like masks or special techniques to adjust camera.

Second (Visualisation):

• There is no consistent way to visualize, your results.
  Jupyter visualization is fine but breaks very easily with more images so we would need to save the output of the final reconstruction. Which I used rec.write(sfm_output_dir) but this only saves the colmap binaries for cameras and does not include the point cloud. !! Dire in need of Rerun.io or at least .gltf so that we can export the result and evaluate independently
• Benchmark results
  I was able to run smerf dataset. But it only saves colmap output and there is nothing else. when python scripts/benchmark.py -d smerf -m minimal is there any of your code applied or is it just for colmap reference?
• Also

python scripts/benchmark.py \
  -d eth3d \ # (default) benchmark dataset
  -m minimal \ # overlap level [/leq5/leq10/leq30/all]
  -s facade \ # scene [/courtyard/electro/...]
  --testset_id 0 \ # id of an ovelrap level (in this case minimal)

is broken and does not work for scene meaning you have to download 20 Gb worth of data be able to do anything. Also Smerf checks if the data is downloaded and skips but ETH dataset download check fails and always redownloads.

If you get 
FileNotFoundError: [Errno 2] No such file or directory: '7z' ->
Don't forget to 
sudo apt install p7zip-full

Summary for visualisation part: I couldn't see any output beside what is actually on the demo.ipynb Normally I would just write my own output and visualiser but there are a lot of functions and classes and benchmark related class that I couldn't navigate to code to actually extract where the point clouds are actually. The code base is brittle and woven together for outside contribution.

Third (Instruction)

Sorry for my bluntness and I don't want to sound rude but your instructions are broken.
First Bug is if you don't use default parameters and pass in parameters like you suggested

# Use default ⚙️
python reconstruct.py \
    --conf sp-lg_m3dv2 \ # see config dir "configs" for other curated options
    --data_dir local/example \ # hosts sfm inputs and outputs when other options aren't specified 
    --intrinsics_path local/example/intrinsics.yaml \ # path to the intrinsics file 
    --images_dir local/example/images \ # images directory
    --cache_dir local/example/cache_dir \ # extraction outputs: depths, normals, matches, etc.
    --extract \ # use ["sky", "features", "matches", "depth", "normals"] to force re-extract
    --verbose 0 

This failles as it assumes to be Path class from pathlib import PATH

Which can be handled by adding

images_dir = Path(images_dir)
cache_dir = Path(cache_dir)

to mpsfm/test/simple.py at line 26 under data_dir conversion
But be ware this will break the demo.ipynb

So you are forces to use python reconstruct.py even though I made the fix that I told you about and was able to run with different configuration it takes the all of the images which is not what I want because I would like to be replicate your result on the first gif under SMERF/Berlin

But you haven't provided the intrinsics you used in that datasets, is that because you extract it from COLMAP? if yes does that mean you guys run first Colmap and determine the actual camera positions then run the depth algorithm on top of that?
Which is still good approach but I just couldn't understand what is going on.

Conclusion

I think the approach is amazing, in fact I was trying to built a similar thing myself which I failed and you seem like succeeded so brillant. And I would really like this approach to work as I think this is the approach we should be taking.

Also the reason I tried so hard I was quite impressed with Mr. Sarlin's previous papers on superglue and lightglue and when I saw the legends like Mr Schoenberger and Mr. Pollefeys I was sure this was bound to be good. But I just couldn't get it working

@Zador-Pataki I think you are the main maintainer as I saw your name in most of the replies, if you would like to consider I would be actually willing to contribute to the project and allocate some time weekly consistently if necessary. But I would need some guidance on where to start.

[Zador-Pataki]

Hey, thanks for the interest in our project and for taking the time to write all this! And I’m sorry to hear you’re struggling with the pipeline. To reproduce your issues, I recreated the environment from scratch and freshly cloned the repo. I’ve opened a mini PR to fix the issues I was able to reproduce — feel free to contribute to it; otherwise, I’ll merge it after we get to the bottom of your problems.

Below is my detailed response:

First (Input)

did you use anything to be able to produce the example outputs?

Yes, the default pipeline extracts sky masks. However, this has only minimal impact.

You wrote earlier in your issue that the reconstruction does not work on your custom dataset. Feel free to send me examples by mail. A few things to check:

• If your intrinsics are incorrect, the pipeline will break.
• MP-SfM relies heavily on the performance of its components. If there are no salient features or extreme perspective changes, and you’re using the SuperPoint+LightGlue variant, the pipeline might fail. LightGlue, in particular, was not trained on such image pairs. If your data is very challenging, I suggest you try sp-mast3r-dense or sp-mast3r which you can set using -c sp-mast3r.

Second (Visualisation)

Jupyter visualization is fine but breaks very easily with more images

The notebook is just meant to show how to convert outputs into a 3D format — not to force you into Jupyter. I’m happy to add utility functions for easier 2D-to-3D conversion. plotly helps with debugging; fig.show() works outside Jupyter too, or you can use fig.write_html(...). You can also check out this previous discussion.

rec.write(sfm_output_dir) but this only saves the colmap binaries for cameras and does not include the point cloud.

As per reconstruction.py, def write also writes depth maps. MP-SfM outputs depth maps (not pointclouds), in addition to the sparse COLMAP-style point cloud. You can unproject these into 3d, and visualize it with your favorite visualizer. For example by converting to .gltf as you proposed, or using plotly.

Dire in need of Rerun.io

Agreed. rerun.io support would be great. I’ll consider adding it when I get the chance.

Benchmark results [...] it only saves colmap output and there is nothing else

The benchmark script is intended for evaluating camera poses only and runs tens, if not hundreds, of reconstructions per dataset. To avoid bloating storage, we avoid caching outputs by default. However, you can feel free to write outputs for example after this line.

when python scripts/benchmark.py [...] is there any of your code applied or is it just for COLMAP reference?

Not sure I understand the question. Can you clarify?

Also [...] is broken and does not work for scene ...

Indeed the bash script should be run without comments. Does that work for you? Does python scripts/benchmark.py -d eth3d -m minimal work for you?

[...] but ETH dataset download check fails and always redownloads.

That doesn’t happen on my setup. ETH3D uses .7z archives (see here). You'll likely need to:

sudo apt install p7zip-full

If your script didn’t extract anything, the check will fail and redownloads will repeat. I can add this to the README if you confirm this was the issue, or you can extend the PR yourself.

Third (Instruction)

your instructions are broken

Fixed in the PR! This works for both reconstruction.py and demo.ipynb. Still broken for you?

But you haven't provided the intrinsics you used in that datasets

SMERF (Zip-NeRF) provides reference COLMAP reconstructions, which include intrinsics. That’s the standard MP-SfM input format: reference COLMAP reconstruction + image dir.

After preparing ETH3D or SMERF via the scripts, you’ll find this format in e.g., mpsfm/local/eth3d/data/electro or mpsfm/local/smerf/data/alameda.

We provide SimpleParser and SimpleTest for easy entry, which creates a reference COLMAP reconstruction for you, but under the hood, MP-SfM follows the format of ETH3DParser and ETH3DTest.

To control image subsets, you can follow mpsfm/local/testsets/eth3d/electro/leq5.yaml and run with:

python scripts/benchmark.py -d eth3d -s electro -m leq5 --testset_id 0 -c sp-mast3r

I hope this helps make using MP-SfM easier. Feel free to continue the discussion, contribute to the PR, or email me if you’d like help applying MP-SfM in your project.

[PoyrazTahan]

The pipeline works, and I'm quite impressed. The full pipeline is definitely on the more complex side, as it's a framework for orchestrating many components within the 3D reconstruction process.

I can also attest that it's working on a wild dataset. I'm still a bit conflicted on where the intrinsics are used exactly, but in general, I got some of the best results using mp-sfm.

Thanks to the authors for this brilliant work!

@Zador-Pataki I'd like to contribute to the project. I can start immediately by improving the visualizations. How would you prefer to coordinate contributions and pull requests?

I'd also like to understand more about the pipeline, especially the feature matching part. When I visualize the matches, they seem to be quite off, which shouldn't be the case since the actual outputs are very good.

[Zador-Pataki]

Glad to hear the problems were resolved. If I find time, I will merge the proposed changes. Being able to flexibly visualize the reconstruction live using runner.io would be great. For contributions, you can just open a pull request. Otherwise, if you have more concrete things you want to discuss, feel free to shoot me an email.