MatSwap: Light-aware material transfers in images

Ivan Lopes¹  ·  Valentin Deschaintre²  ·  Yannick Hold-Geoffroy²  ·  Raoul de Charette^1
1Inria    ²Adobe Research

🎉 MatSwap received the Best Paper Honorable Mention Award at EGSR 2025! 🎉

✨ Overview

MatSwap transfers materials to designated surfaces in an image realistically. Such a task is non-trivial due to the large entanglement of material appearance, geometry, and lighting in a photograph. In the literature, material editing methods typically rely on either cumbersome text engineering or extensive manual annotations requiring artist knowledge and 3D scene properties that are impractical to obtain. In contrast, we propose to directly learn the relationship between the input material, as observed on a flat surface, and its appearance within the scene, without the need for explicit UV mapping. To achieve this, we rely on a light- and geometry-aware diffusion model. We train a large-scale pre-trained text-to-image model for material transfer using our synthetic dataset, preserving its strong priors to ensure effective generalization to real images. As a result, our method seamlessly integrates a desired material into the target location in the photograph while retaining the identity of the scene.

📦 Setup

• Cloning source: after cloning this repository with:

  git clone https://github.com/astra-vision/MatSwap.git
  

  You will need to pull the third-party code as well from the RGBX repository, it is already registered as a submodule, simply run:

  git submodule update --init --recursive
  

• Requirements: this repository has minimal requirements, essentially torch and diffusers as primary libraries. MatSwap was last tested on Python v3.11.5. Depending on your CUDA version, you may need to use a different cuda toolkit version compatible torch package. Please refer to useful page (make sure the cuda version displayed is lower that the one shown when displaying nvidia-smi). It is recommended to use conda, pyenv, or any other form of library manager to isolate this installation:

  pip install torch==2.3.0 torchvision==0.18.0 --index-url https://download.pytorch.org/whl/cu121
  pip install diffusers==0.30.3 transformers==4.41.2 accelerate==0.31.0
  pip install opencv-python
  

• Checkpoint: the model is implemented as a diffusers pipeline and can be downloaded directly from the hub. Overall it works as:

  pipe = MatSwapPipeline.from_pretrained('ilopes/matswap')
  

  The trained IP-Adapter weight will be automatically loaded onto the pipeline and cached on your machine. If you do not have network access when running the code, you can alternatively install the checkpoints locally yourself using git-lfs: git clone https://huggingface.co/ilopes/matswap.

🚀 Quick Start

We provide an example code in demo.py to showcase how the pipeline can be used to perform material inpainting. This repository also contains samples in /data, we do not own any of its content. The images were obtained from the library Unsplash and are open source. The material images were sampled from MatSynth.

python demo.py \
    --image data/image/cca59bfc-image_050.png \
    --mask data/mask/cca59bfc-image_050_C.png \
    --material data/material/st_fabric_063_001.png

The code will automatically generate the additional scene descriptors (normals and irradiance) using the third-party code (RGBX), and place them under data.

Additionally we provide the following list of parameters.
To control the hue and scale of the material use --scale or --hue.

Argument	Type	Default	Description
--image	Path	—	Target image to modify
--mask	Path	—	Binary mask defining the replacement area
--material	Path	—	Source material texture image
--scale	int	0	Material scale exponent s (crop size / 2^s)
--hue	int	0	Hue shift (0–180) applied to the material
--seed	int	0	Random seed for reproducibility
--guidance_scale	float	3.0	Classifier-free guidance scale

Results are saved to outputs/ as {image_stem}_{material_stem}.png.

🙏 Acknowledgements

This project would not have been possible without the HuggingFace library suite (transformers, diffusers, accelerate) and Blender. We thank the authors of MatSynth, IP-Adapter, and RGBX for its valuable contributions. The work of the first author was funded by the French Agence Nationale de la Recherche (ANR) with project SIGHT, ANR-20-CE23-0016. All experiments were performed using GENCI-IDRIS HPC resources (Grants AD011014389R1, AD011012808R3) and HPC infrastructure at Inria Paris. Caution

🔑 Licence

MatSwap is released under MIT License.

Caution

The diffusion checkpoint used (Stable Diffusion v1-5) is licensed under the CreativeML Open RAIL-M license. Please note that the license includes specific obligations (such as marking modified files) that are not covered by the MIT license of this repository. Users are responsible for reviewing and complying with both licenses when using or modifying the model. See model license for details.

📄 Citation

If you use our work, please cite:

@inproceedings{lopes2025matswap,
  author    = {Lopes, Ivan and Deschaintre, Valentin and Hold-Geoffroy, Yannick and de Charette, Raoul},
  title     = {MatSwap: Light-aware material transfers in images},
  booktitle = {Computer Graphics Forum},
  year      = {2025},
}