Feature request: Add background removal and precise mode + raise point limit

[dokipen3d]

Hiya

I've been hacking the plugin to be able to do background removal before romav2 matching. I also am working with sparse views of an object (~8 views) so upping the point count and accuracy gave me really good densification results.

I'll see if I can do a pull request. Also would be nice if we can export a dense colmap scene using the triangulated points.

[shadygm]

Hey @dokipen3d , thanks for taking a look at the project (:

Usually the expectation for object-only reconstruction is that your images are already masked beforehand, that way the plugin only reconstructs the non-masked regions (that’s also how I’ve been doing it so far). So I'm not 100% sure this plugin should take care of the background removal/masking part of stuff. Might be good to have as its own plugin.

Regarding exporting a dense COLMAP scene, I don’t think it’s strictly necessary in the plugin right now, since you can already right-click the point cloud in the scene graph and save it to disk. But if you have a specific workflow in mind where a full COLMAP export would help, feel free to share it and I can take a look.

Also curious, what exactly did you tweak to get the higher point count / accuracy?

[dokipen3d]

in matcher.py, I added a check for a new "precise" setting that matches RomaV2's own precise setting.

I also changed the matches_per_ref range to go up to 128,000 (I might even try higher).

        del mode  # Legacy arg kept for API compatibility.
        self.device = torch.device(device)
        torch.set_float32_matmul_precision("highest")
        self.model = RoMaV2(RoMaV2.Cfg(compile=False))
        if setting == "high":
            self.model.H_lr = 640
            self.model.W_lr = 640
            self.model.H_hr = 960
            self.model.W_hr = 960
            self.model.bidirectional = True
        elif setting == "precise":
            self.model.H_lr = 800
            self.model.W_lr = 800
            self.model.H_hr = 1280
            self.model.W_hr = 1280
            self.model.bidirectional = True
            self.model.threshold = None
            self.model.balanced_sampling = True
        else:
            self.model.apply_setting(setting)```

[dokipen3d]

The idea of being able to export colmap scene is that if we export a proper images.txt and points3D.txt we can use other libraries that require the points to be tied to their cameras. I can export the ply directly, but they are just a big cloud of points and the cameras don't know which ones correspond to them.

One that springs to mind is...
https://github.com/zju3dv/Murre

[shadygm]

ah these are all good points. I'd be happy for any incoming PR that adds these features. Looking forward to it (: