Using masks for training

[MotivaCG]

I hope you don't mind me citing you here, @pierotofy and @NotCompsky.

I followed pull request #154, fixed some minor issues, and optimized parts of the code to calculate the ROI of the masked image to speed up training.

It actually works, but it generates many floaters near the correct surface, many more than without the mask. On the other hand since the trained is focused on the masked areas they look more sharper and detailed.

I would like to use this discussion to share ideas and continue this development. At the moment, I noticed that in GSplat they made a modification in the rasterizer to support alpha. I’m not sure if OpenSplat was already based on GSplat with that modification integrated or if it needs to be added: https://github.com/nerfstudio-project/gsplat/pull/70/files

Regards, Víctor

[pierotofy]

You'll need to check the source code and see if the relevant parts are in OpenSplat. Cool stuff you're working on!

[NotCompsky]

Thanks for the interest in my PR!

My PR worked fine actually - I've used it for a few renderings now with no visible flaws - but I retracted it because it broke OpenSplat for non-mask renders xD (and I was a bit embarrassed so I had hoped nobody would see it xD)

I'm going to be busy for the meantime, but you can clone my fork, compile it, and just use it for masked renderings.

It should be easy to find and fix what is causing the bug, but I tried for ages and gave up. It must be a blind spot for me.

It actually works, but it generates many floaters near the correct surface, many more than without the mask.

Yeah, that's expected - there's no code to restrict where it places the splats, only where it calculates the difference.

It would actually be possible to restrict where it places the splats - but that would be a bit more work. :)

---

Edit: I might have replied to the wrong person, sorry @MotivaCG

[jared-krauss]

Oooh this could be relevant to my work, as with my chaotic data set (e.g. city at night with moving cars, etc.), I'm masking quite a bit of the images, and getting very strange floaters.

So, if I understand correctly, this will help specifically render the parts of an image that are not masked out?

If so, would be very helpful! Now I just have to learn how to clone a fork and compile it to my mac. I am a total noob, so even just getting Colmap and Opensplat running was a month long chore haha but this is very intriguing and useful for me.