Adaptive Spatio-Temporal 3D Gaussian Splatting for Scenes with Oscillatory Motion

Petros Tzathas, Jeffrey Hu, Andréas Meuleman, Guillaume Cordonnier, and George Drettakis
| Webpage | Full Paper | Other GRAPHDECO Publications | NERPHYS project page |

This repository contains the official implementation of the paper "Adaptive Spatio-Temporal 3D Gaussian Splatting for Scenes with Oscillatory Motion", which can be found here.

Abstract: Our goal is to reconstruct scenes with stochastic, incoherent motion such as leaves moving in the wind, that can be particularly challenging because of small objects with similar appearance that move independently. Previous dynamic 3D Gaussian Splatting solutions either represent motion implicitly with neural networks achieving good quality but lower framerate, or explicitly with a function, often with higher training times and lower quality. To overcome these limitations, we propose an explicit method that introduces adaptive space-time densification and smoother optimization. We introduce a new densification approach based on error moments that are used to guide primitive splitting, and we adaptively refine the number of keyframes used based on the variance of error. We observe that dynamic reconstruction from monocular video is hard for standard optimization pipelines. To counter this, we introduce a weighted Adam approach that improves results based on primitive visibility. Finally, to handle the hard case of independent motion of similar-looking objects, we introduce an image-driven as-rigid-as-possible regularization. Our method has higher quality than previous explicit solutions, and has significantly higher framerate for rendering.

Funding and Acknowledgments

This work was funded by the European Union, European Research Council (ERC) Advanced Grant NERPHYS, 101141721 https://project.inria.fr/nerphys. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). The authors would also like to thank Adobe and NVIDIA for software and hardware donations.

Overview

The codebase is organized as follows:

dynamic-gaussian-splatting/
├── scripts/                    # Entry points
│   ├── train.py                # Training
│   ├── render_model.py         # Render test views from a trained model
│   ├── evaluate_renders.py     # Compute PSNR/SSIM/LPIPS metrics
│   ├── inspect_checkpoint.py   # Visualize a trained checkpoint
│   ├── compute_keyframe_stats.py
│   └── measure_fps.py
│
├── dynamic_gaussians/          # Core library
│   ├── training_loop.py        # Main training loop
│   ├── weighted_adam.py        # Visibility-weighted Adam optimizer
│   ├── models/                 # Gaussian model implementations
│   │   ├── gaussian_model.py               # Abstract base class
│   │   ├── keyframe_gaussian_model.py      # Dynamic keyframe model
│   │   └── keyframe_gaussian_model_taichi.py  # Taichi-accelerated interpolation
│   ├── gaussian_renderer/      # Differentiable rasterization wrapper
│   ├── point_clouds/           # Point cloud data structures
│   ├── losses/                 # Loss functions (image, geometric, ARAP)
│   ├── densification/          # Adaptive densification strategies and splitting
│   ├── scene/                  # Scene/camera loading (COLMAP, NeRF formats)
│   ├── arguments/              # Hyperparameter definitions
│   └── utils/                  # Geometry, SE(3), SH, and other utilities
│
└── cuda/                       # Custom CUDA extension (diff-gaussian-rasterization)

The main training logic lives in dynamic_gaussians/training_loop.py, which is invoked by scripts/train.py. The motion model is implemented in models/keyframe_gaussian_model.py using explicit keyframes with linear velocity interpolation and relative quaternion rotations. Adaptive densification is handled by the densification/ module, using error moment statistics to guide primitive splitting and keyframe refinement.

Setup

Cloning the Repository

The repository includes glm as a submodule, please clone it with the --recursive option:

git clone git@github.com:graphdeco-inria/adaptive-spatio-temporal-gaussians.git --recursive

# or

git clone https://github.com/graphdeco-inria/adaptive-spatio-temporal-gaussians --recursive

Alternatively the submodule can be initialized after cloning with:

git submodule update --init --recursive

Creating the environment

We provide an environment.yml which can be used to create a conda environment:

conda env create --file environment.yml
conda activate dynamic-3dgs

Afterwards, the cuda extension needs to be build. On Windows the environment variable DISTUTILS_USE_SDK needs to be set

CMD:

SET DISTUTILS_USE_SDK=1

PowerShell:

$env:DISTUTILS_USE_SDK=1

We also recommend running these and the following in the Developer Command Prompt / Powershell profiles that add the compiler to the path.

Then for building the extension:

pip install ./cuda/diff-gaussian-rasterization/ --no-build-isolation

(The flag --no-build-isolation is required in more recent versions of pip in order for it to be able to use the python packages installed in the environment, pytorch notably.)

A note on Visual Studio

At the time of development, CUDA 12 was not working with Visual Studio 2026, so it is advised to use Visual Studio 2022. For some reason Microsoft has made it hard to get previous versions of Visual Studio but you can get still get the 2022 version here.

Running scripts

To see the available options of a script as well as their function, simply run:

python <path>/<to>/<script.py> --help

In order for the library code to be visible to the scripts, we recommend adding the the repository directory to PYTHONPATH. This can probably be done in the IDE of your choice. For example in VSCode the following can be added to a configuration in launch.json:

"env": {
    "PYTHONPATH": "${env:PYTHONPATH};${workspaceFolder}",
}

It can also be done for a specific command. For example in PowerShell:

$env:PYTHONPATH="${env:PYTHONPATH};${pwd}.Path"; python ...

Training

To train a scene you can run:

python scripts/train.py -s <path>/<to>/<dataset> -m <path>/<to>/<output>/<directory> --cache_images

Some important options

• --gui: Launches the GUI for visualizing the model as it trains
• --eval: Splits the dataset in train and test for evaluation
• --minimal_reporting: Avoids large tensorboard files

Large datasets

Some datasets, for example N3DV which is multiview, might not fit in memory. In this case the --cache_images option can be omitted and the images will be reloaded as they are used. Additionally, multi-process dataloading can be used by specifying --num_workers N, but in this case --data_device cpu should also be specified for reasons detailed in the link given above.

Evaluation

We provide scripts for evaluating a model:

python scripts/render_model.py -m <path>/<to>/<training>/<output>/<directory> -o <path>/<to>/<evaluation>/<directory>

python scripts/evaluate_renders.py --workdir <path>/<to>/<evaluation>/<directory>

python scripts/measure_fps.py -m <path>/<to>/<training>/<output>/<directory>

python scripts/compute_keyframe_stats.py -m <path>/<to>/<training>/<output>/<directory>

We also provide the notebook summarize_results.ipynb as a convenience for parsing and displaying the results of the evaluation.

Model inspection

A trained model can be visualized with the GUI as follows:

python scripts/inspect_checkpoint.py -m <path>/<to>/<training>/<output>/<directory>

BibTeX

@article{tzathas2026adaptive,
  title     = {Adaptive Spatio-Temporal {3D} Gaussian Splatting for Scenes with Oscillatory Motion},
  author    = {Tzathas, Petros and Hu, Jeffrey and Meuleman, Andr\'{e}as and Cordonnier, Guillaume and Drettakis, George},
  journal   = {Computer Graphics Forum},
  volume    = {45},
  number    = {2},
  year      = {2026},
  publisher = {Eurographics - The European Association for Computer Graphics and John Wiley \& Sons Ltd.}
}