Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

Yiming Huang *, Beilei Cui *, Long Bai *, Ziqi Guo, Mengya Xu, Hongliang Ren

Medical Image Computing and Computer Assisted Intervention (MICCAI) 2024

|| Paper || Arxiv ||

TODO

• EndoNeRF
• StereoMIS
• Pretrained checkpoints

Environments

We build the Python environment using Anaconda:

1. Install the CUDA toolkit on ubuntu following this, and then:

export PATH=/usr/local/cuda-11.8/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda-11.8

2. Install the Python environment

git clone https://github.com/lastbasket/Endo-4DGS.git
cd Endo-4DGS
git submodule update --init --recursive
conda create -n ED4DGS python=3.8
conda activate ED4DGS

pip install -r requirements.txt
pip install -e submodules/diff-gaussian-rasterization-depth
pip install -e submodules/simple-knn
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118
pip install torchmetrics

Datasets

We used two datasets for training and evaluation.

EndoNeRF

We used EndoNeRF dataset provided by Yuehao Wang.

StereoMIS

We used StereoMIS dataset from Michel Hayoz

The data structure is as follows:

./data
├── endonerf
│    ├── cutting_tissues_twice
│    │    └── ...
│    ├── pulling_soft_tissues
│    │    └── ...
│    └── ...
└── stereomis
    └── ...

How to prepare stereomis:

# 1. unzip StereoMIS_0_0_1.zip to StereoMIS_0_0_1
# 2. run the script
bash prepare_stereomis.sh

# 3. arrange the files as
./data
└── stereomis
    └── split_1
    │    │    └── images
    │    │    └── poses_bounds.npy
    │    │    └── ...

# 4. you may also visualize the trajectory using
cd stereomis
python visualize_traj.py

Preparing the Pretrained Depths from Depth-Anything

Download depth_anything_vits14.onnx and place in: ./submodules/depth_anything/weights/depth_anything_vits14.onnx

sh prepare_depth.sh

Training

sh train.sh

More configurations can be found in arguments/$DATASET.py

Evaluation

# Render the testing result
sh render.sh
# Evaluation
sh eval.sh

Inferring the pretrained Gaussians

Download pretrained_gaussians and unzip in the project directory:

# Render the testing result e.g. endonerf/pulling
python render.py --model_path pretrained_gaussians/endonerf/pulling --pc --skip_video --skip_train --configs arguments/endonerf.py

Related Works

Welcome to follow our related works:

• SurgTPGS: Vison-Language Surgical 3D Scene Understanding
• Endo-4DGX: Robust Endoscopic Gaussian Splatting with Illumination Correction
• Endo2DTAM: Gaussian Splatting SLAM for Endoscopic Scene

Cite

@inproceedings{huang2024endo,
  title={Endo-4dgs: Endoscopic monocular scene reconstruction with 4d gaussian splatting},
  author={Huang, Yiming and Cui, Beilei and Bai, Long and Guo, Ziqi and Xu, Mengya and Islam, Mobarakol and Ren, Hongliang},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={197--207},
  year={2024},
  organization={Springer}
}

Acknowledgement

Thanks the authors for their works: StereoMIS, diff-gaussian-rasterization-depth, EndoNeRF, 4DGaussians, Depth-Anything-ONNX.