Image masking

[robertguetzkow]

I'd like to suggest image masking as a feature for a future version of Meshroom. In case this would be implemented, masked areas should not be used for keypoint detection nor for depth maps and texture generation. A simple import of masks in the form of binary images would be enough, a build in editor is not necessary.

Thank you for developing this great photogrammetry software!

[finnbear]

@robertguetzkow Can I ask how you would create your binary masks? I don't know how I would do that right now so I would want a built in editor.

[octupe]

You could just paint in some rough mask per image in GIMP or photoshop and save it as a PNG using the image name it applies to with a predetermined extension or suffix.

[robertguetzkow]

@finnbear exactly as @octupe described. Sorry been on vacation, that's why I didn't respond sooner.
GIMP, Krita, Affinity Photo, Photoshop or any other image editing software that allows to use layers.

[fabiencastan]

I agree that this would be a good feature.

In the meantime, a simple workaround is to apply your mask on the input pixel (RGB=0). There is no drawback in doing that. You just need to be careful to keep your image metadata (if your image editing software doesn't preserve metadata, you can transfer metadata from one image to another with exiftool).

[robertguetzkow]

@fabiencastan wouldn't that workaround select edges in the RGB=0 area as features, therefore possibly mismatching images when masking multiple objects/areas?

Yes, metadata needs to be preserved for focal length calculation. Exiftool is very useful if you need to manually add, edit or copy exif data. I definitely recommend it.

[fabiencastan]

Yes sure but in practice it works quite fine.
I have always used soft-edges on my masks.

[natowi]

I think a masking node could be based on G'MIC.

[ChemicalXandco]

If there was to be built-in masking, a hierarchical agglomerative clustering procedure that expands the background and foreground regions from the initial strokes seems to be the best solution.

[zfarm]

Sorry Im a bit new. Does this entail pulling each photo into an editor and painting out what isnt necessary for the scan?
I agree a bounding volume you could define once the point cloud was generated would save a lot of time .

[robertguetzkow]

@zfarm yes that was my idea. It has the benefit that there are already plenty image editors out there that are stable and it would greatly reduce the necessary work to integrate this feature. Integrating a mask editor into Meshroom seems overkill to me.

[natowi]

@robertguetzkow Of course you can apply masks to a few images or paint out unwanted areas by hand - but what about datasets with 100+ images? GMIC can extract the foreground and create reject masks - but not in bulk.
I was thinking more of Image Segmentation like this to automatically identify and select the object in multiple images and create masks.

@zfarm Yes, a bounding volume/Mask (as a tool in the 3D Viewer?) for the point cloud would be nice.
Maybe PCL (->github) could be useful here.

[robertguetzkow]

@natowi sure I can think of several more automated and sophisticated approaches, but somebody has to implement it as well. I was just being pragmatic. Having any masking at all would be an improvement.
In case somebody wants to try your segmentation approach, I'd highly suggest to not rely on automated processing alone. There should always be a way to make manual adjustments to the masks.

[natowi]

I'd highly suggest to not rely on automated processing alone

@robertguetzkow I agree
When I have some time, I´ll look for a suitable library or tool we could use for this task and think about a way to integrate it in the Meshroom workflow. (This is a good starting point)