Skip to content

README.md

Latest commit

 

History

History
184 lines (127 loc) · 9.81 KB

README.md

File metadata and controls

184 lines (127 loc) · 9.81 KB

Intrinsic decomposition and editing of 3D Gaussian splats

Alexandre Lanvin, Jeffrey Hu, Simon Lucas, Adrien Bousseau, George Drettakis

| Webpage | Full paper | Other GRAPHDECO Publications | NERPHYS project page | Datasets | Pretrained |

teaser figure

This repository contains the official implementation of the I3D 2026 paper "Intrinsic decomposition and editing of 3D Gaussian splats", which can be found here.

Abstract: Intrinsic decomposition — which expresses image colors as the product of diffuse albedo and shading, possibly augmented with view-dependent residuals — has a long history in image editing as it enables the modification of object colors and textures without altering lighting. We extend intrinsic decomposition to radiance fields represented with Gaussian splatting by proposing solutions to three key aspects of such decomposition. First, we describe how to model the intrinsic decomposition as independent sets of Gaussian primitives, which allows each set to adapt to the characteristics of the layer it represents. Second, we present an optimization procedure guided by data-driven predictions to disentangle multi-view photographs of a scene into the aforementioned intrinsic sets. Finally, we provide an editing workflow where users modify the texture of planar surfaces simply by modifying the albedo of that surface in one image. Capturing this edit within the intrinsic radiance field allows re-rendering of the edited scene with plausible lighting under arbitrary viewpoints.

BibTeX

@article{lanvin:hal-05602203,
  TITLE = {{Intrinsic decomposition and editing of 3D Gaussian splats}},
  AUTHOR = {Lanvin, Alexandre and Lucas, Simon and Hu, Jeffrey and Bousseau, Adrien and Drettakis, George},
  URL = {https://hal.science/hal-05602203},
  JOURNAL = {{Proceedings of the ACM on Computer Graphics and Interactive Techniques}},
  PUBLISHER = {{ACM}},
  YEAR = {2026},
  MONTH = May,
  DOI = {10.1145/3804495},
  KEYWORDS = {CCS Concepts: Computing methodologies $\rightarrow$ Rasterization Neural networks Reflectance modeling ; CCS Concepts: ; Computing methodologies $\rightarrow$ Rasterization ; Neural networks ; Reflectance modeling},
  PDF = {https://hal.science/hal-05602203v1/file/paper_authors_version.pdf},
  HAL_ID = {hal-05602203},
  HAL_VERSION = {v1},
}

Funding and Acknowledgments

This work was funded by the European Union, European Research Council (ERC) Advanced Grant NERPHYS, 101141721 https://project.inria.fr/nerphys. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). The authors would also like to thank Adobe and NVIDIA for software and hardware donations.

Setup

Tested on windows with nvcc 12.5 and Ubuntu 24.04 with gcc 12 and cuda 12.1

Cloning the Repository

The repository includes multiple submodules, please clone it with the --recursive option:

git clone --recursive https://github.com/graphdeco-inria/intr_decomp_3dgs.git
cd intr_decomp_3dgs

If you cloned without --recursive, run git submodule update --init --recursive.

Creating the conda environment

SET DISTUTILS_USE_SDK=1 # Windows only

conda create -n 3dgs_decomp python==3.11
conda activate 3dgs_decomp

# Install a pytorch+cu version that has same major version as your installed nvcc (probably 11x or 12x)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# Install submodules
pip install third_party/fused_ssim third_party/simple-knn ./diff-gaussian-rasterization --no-build-isolation 

# Install remaining pip dependencies
pip install -r requirements.txt

Running scripts

We re-used most of the existing code from the original 3DGS implementation and provide a single training script to reconstruct the three sets of gaussians representing albedo, shading and residual fields. An input scene must contain a colmap sparse/0 folder, images, albedo maps and inverse depth maps.

Training

With the 3dgs_decomp conda env activated, run the following command to train a full decomposition:

python train.py -s <source_path> -m <model_path> \
    --albedo albedo --depths depths \
    --eval --enable_viewer

Training runs three sequential stages, each writing to <model_path>/point_cloud/iteration_<N>/:

  1. Albedo (--albedo_iterations, default 30k) → albedo.ply
  2. Shading (--shading_iterations, default 30k) → shading.ply (the albedo from step 1 is frozen)
  3. Residual (--residual_iterations, default 30k) → residual.safetensors (both layers above frozen). Residual gaussians are seeded by finding pixels where gt - albedo*shading > 0.1 and unprojecting 1% of them at the GT depth — see train.py:62.
New command Line Arguments

--albedo / -a

Subfolder with pseudo-GT albedo. (albedo by default).

--srgb_input

Treat images/ as sRGB and convert to linear on load. (OFF by default).

--clamp_albedo

Clamp the GT albedo to [0,1] during the albedo stage. (ON by default).

--enable_viewer

Launch the viewer in a background thread, live-attached to the optimization. (OFF by default).

Outputs

After training, <model_path>/ will contain:

<model_path>/
├── cfg_args                    # Namespace dump of the ModelParams used; the viewer reads this back.
├── input.ply                   # copy of sparse/0/points3D.ply (the init point cloud)
├── point_cloud/
    ├── iteration_7000/
    │   ├── albedo.ply
    │   ├── shading.ply
    │   └── residual.safetensors
    └── iteration_30000/
        ├── albedo.ply
        ├── shading.ply
        └── residual.safetensors

Viewer

We reused graphdecoviewer and added widgets to enable editing, gaussian_viewer.py is a unified PyOpenGL + imgui viewer with three modes:

Mode Use it when Editing Args
local Inspecting / editing a trained model on disk. Yes — selection-box → texture plane → multi-view re-optimization. model_path iter
server Same as local but also accepts a remote client. No model_path iter --ip --port
client Lightweight UI that attaches to a running training session (train.py --enable_viewer) or to a server. No --ip --port

Launching

# Open a trained model at iteration 30000:
python gaussian_viewer.py local <model_path> 30000

# Attach to a training session running on the same machine:
python train.py -s <source_path> -m <model_path> --enable_viewer
# (the viewer opens automatically in a background thread)

# Remote attach:
python gaussian_viewer.py server <model_path> 30000 --ip 0.0.0.0 --port 6009
python gaussian_viewer.py client --ip <server-ip>  --port 6009

local and server both read <model_path>/cfg_args to recover the original ModelParams / PipelineParams, then load whichever of albedo.ply, shading.ply, residual.safetensors exist under point_cloud/iteration_<iter>/ (gaussian_viewer.py:204).

Editing

Given a reconstructed 3D intrinsic decomposition using our training script, we provide an user interface to edit the albedo gaussians using an uploaded image. To be able to edit the reconstruction make sure the current render_mode is set to Albedo: Tuto edit

Evaluation

If a model was erconstructed using --eval flag then the evaluate.py script can be used to render test views and compute metrics PSNR, SSIM and LPIPS using input images:

python evaluate.py -s <source_path> -m <model_path > --iteration <nb_iterations>

6. Troubleshooting

  • Colmap camera model not handled — your COLMAP project uses a distortion model. Run colmap image_undistorter --input_path ... --output_path ... --output_type COLMAP first.
  • Depth regularizer has no effect — check that --depths <dir> points to a folder that exists and shares file stems with images/. If you provided a 16-bit PNG depth, also generate sparse/0/depth_params.json; without it, scales are taken as 1.0.
  • Viewer opens black — confirm the loaded iteration has at least one of albedo.ply / shading.ply / residual.safetensors under point_cloud/iteration_<iter>/.
  • SparseGaussianAdam ImportError — rebuild diff-gaussian-rasterization against the variant that exports SparseGaussianAdam.