Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering

Unofficial implementation of Scaffold-GS as a nerfstudio extension.

teaser.mp4

Additionally, we include depth+normal optimization from RaDe-GS.

normal.mp4

GSDF: 3DGS Meets SDF for Improved Neural Rendering and Reconstruction

Now includes an implementation of GSDF.

The below results are a work-in-progress and were trained without any hyperparameter fine-tuning (which NeuS is very sensitive to).

RGB	Depth	Normal

Note: there are several differences when compared to the reference implementation, as we build off of the nerfstudio implementation of NeuS.

Neus-Acc: NeuS with empty space skipping based on nerfacc

Now includes a port of NeuS-Acc from SDFStudio, ported for recent versions of nerfstudio and gsplat.

Note: Neus-Acc implementation is currently experimental.

Status

• Functionality
  • Correctly rendering and training
  • Anchor adjustment
  • Debugging
• Visualization
  • Fix viser functionality
  • ...
• Extras
  • GSDF implementation
  • GSDF mesh export
  • [~] NeuS-Acc port
  • Scaffold-GS mesh export
  • ...

Environment

This repository has been tested on the following environment. Other similar configurations should also work.

pop-os 22.04
cuda 11.8
gcc 10.5
python 3.10.16
numpy 1.26.4
torch 2.1.2+cu118
torchvision 0.16.2+cu118

Registering with Nerfstudio

Ensure that nerfstudio has been installed according to the instructions. Clone or fork this repository and run the commands:

conda activate nerfstudio
cd scaffold-gs-nerfstudio/
pip install -e .
ns-install-cli

You must also install the correct torch_scatter for your environment (https://pytorch-geometric.com/whl/torch-2.1.2%2Bcu118.html)

Running Scaffold-GS

This repository creates a new Nerfstudio method named "scaffold-gs". To train with it, run the command:

ns-train scaffold-gs --data [PATH]

GSDF can be run via the following command:

# for indoor scenes
ns-train gsdf --data [PATH] --pipeline.model.sdf-field.inside-outside True
# for object scenes
ns-train gsdf --data [PATH] --pipeline.model.sdf-field.inside-outside False

Unbounded scenes are not supported, but might work with --pipeline.model.sdf-field.inside-outside True.

NeuS-Acc can be run via the following command:

ns-train neus-acc --data [PATH]

Acknowledgements

Thanks to the authors of Scaffold-GS for the reference implementation, which we build upon heavily.

Thanks to the authors of GSDF for the reference implementation of GSDF, which we build upon heavily.

Thanks to the authors of SDFStudio for their implementation of NeuS-acc.

Thanks to the authors of Splatfacto-360 for their code converting nerfstudio cameras to the COLMAP format.

Thanks to the authors of RaDe-GS for their differentiable depth + normal rasterization code.