mrSDMUniPS

A Meshroom node integration for SDM-UniPS: Scalable, Detailed, and Mask-Free Universal Photometric Stereo (CVPR2023 Highlight).

Overview

This project provides a Meshroom node wrapper for the SDM-UniPS neural network, enabling seamless integration of universal photometric stereo reconstruction into the Meshroom pipeline. SDM-UniPS can recover intricate surface normal maps and BRDF parameters (albedo, roughness, metallic) from images captured under unknown, spatially-varying lighting conditions.

Key Features:

• Universal Photometric Stereo: Works with arbitrary lighting conditions without calibration
• Mask-free Processing: No object masks required (though optional masks are supported)
• High-quality Results: Rivals 3D scanner quality for surface normal estimation
• BRDF Recovery: Estimates albedo, roughness, and metallic parameters
• Scalable Processing: Handles high-resolution images efficiently
• Meshroom Integration: Seamless workflow with AliceVision SfMData format

Installation

1. Clone this repository into your Meshroom nodes directory
2. Install the required dependencies:

cd /path/to/mrSDMUniPS
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Download the pretrained models from here and extract them to your checkpoint directory

Usage

Input Requirements

The SDMUniPS node accepts:

• SfMData file: AliceVision SfMData containing images and camera poses
• Optional masks: Directory with pose-specific masks or alpha channel images
• Multiple images per pose: Images captured under different lighting conditions

Node Parameters

• Target Output: Choose between normal maps, BRDF parameters, or both
• Max Image Resolution: Maximum resolution for processing (default: 4096)
• Max Images per Pose: Limit number of images processed per pose (default: 10)
• Scalable Mode: Enable for high-resolution processing with reduced memory usage
• Enable Cropping: Automatic cropping based on mask bounding boxes
• Checkpoint Path: Path to the pretrained model files

Output Files

The node generates:

• Normal Maps: Surface normals in camera coordinate system (<POSE_ID>_normals.png)
• Albedo Maps: Base color maps (<POSE_ID>_albedo.png)
• Roughness Maps: Surface roughness parameters (<POSE_ID>_roughness.png)
• Metallic Maps: Metallic parameters (<POSE_ID>_metallic.png)
• SfMData files: Updated SfMData with output image references

TODO

• Update camera intrinsics after cropping operations
• Fix output generation bug

References

Original Paper:

@inproceedings{ikehata2023sdmunips,
  title={Scalable, Detailed and Mask-free Universal Photometric Stereo},
  author={Satoshi Ikehata},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}

Links:

• Original Repository
• Paper (ArXiv)
• Author's Page
• Pretrained Models
• Sample Data