Yupeng Zhang, Dezhi Zheng, Ping Lu, Han Zhang, Lei Wang, Liping xiang, Cheng Luo, Kaijun Deng, Xiaowen Fu, Linlin Shen, Jinbao Wang
| Full Paper | PRCV 2025 |
Figure 1: Overall framework of LabelGS.
Abstract: 3D Gaussian Splatting (3DGS) has emerged as a novel explicit representation for 3D scenes, offering both high-fidelity reconstruction and efficient rendering. However, 3DGS lacks 3D segmentation ability, which limits its applicability in tasks that require scene understanding. The identification and isolation of specific object components is crucial. To address this limitation, we propose Label-aware 3D Gaussian Splatting (LabelGS), a method that augments the Gaussian representation with object label.
LabelGS introduces cross-view consistent semantic masks for 3D Gaussians and employs a novel Occlusion Analysis Model to avoid overfitting occlusion during optimization, Main Gaussian Labeling model to lift 2D semantic prior to 3D Gaussian and Gaussian Projection Filter to avoid Gaussian label conflict.
Our approach achieves effective decoupling of Gaussian representations and refines the 3DGS optimization process through a random region sampling strategy, significantly improving efficiency. Extensive experiments demonstrate that LabelGS outperforms previous state-of-the-art methods, including Feature-3DGS, in the 3D scene segmentation task. Notably, LabelGS achieves a remarkable 22
We provide 4 verifiable datasets (3d_ovs, nerf_llff_data, lerf_mask, 360_v2) for evaluation. Each dataset contains two folders: images and segmentations. The example of the dataset structure is as follows. The dataset folder is under the project folder LabelGS.
dataset/
│
├─ 3d_ovs/
│ │
│ ├─ sofa/
│ │ ├─ images/ # Raw RGB scene images (for 3D Gaussian reconstruction)
│ │ └─ segmentations/ #2D semantic masks (for 3D object segmentation evaluation)
│ │
│ ├─ room/
│ │ ├─ images/
│ │ └─ segmentations/
│ │
│ └─ ...
│
└─ 360_v2/
└─...
First, clone the code repository and install dependencies:
git clone https://github.com/garrisonz/LabelGS.git --recursive
cd LabelGS
conda create -n labelgs python=3.8
conda activate labelgs
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install plyfile==1.0.3
pip install tqdm scipy wandb opencv-python scikit-learn lpips imageio scikit-image matplotlib
pip install pulp pycocotools segment_anything timm supervision
pip install dearpygui # visualization
pip install submodules/labelgs-rasterization/
pip install submodules/simple-knnNext, download required foundation models (for depth estimation, segmentation, etc.):
mkdir -p checkpoints
# download for Depth_anything
wget -P checkpoints/ https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth
# download for DEVA
wget -P ./saves/ https://github.com/hkchengrex/Tracking-Anything-with-DEVA/releases/download/v1.0/DEVA-propagation.pth
wget -P ./saves/ https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
Run the following scripts sequentially to preprocess the dataset (take 3d_ovs/room as an example):
# obtain depth maps
python preprocess/unocclusion_mask/depth_estimation.py --dataset_path dataset/3d_ovs --scene room
# obtain multi-view masks
python preprocess/consistent_mask/demo_automatic_all.py --dataset_name 3d_ovs --scene room
python preprocess/consistent_mask/uni_mask.py --dataset_path dataset/3d_ovs --scene room
# calculate occlusion relationship
python -m preprocess.unocclusion_mask.get_occlude_mapping --dataset_path dataset/3d_ovs --scene room
# estimate camera poses by colmap
python preprocess/colmap_tool/run_colmap.py --base_dir dataset/3d_ovs/room
Train the Label-Aware 3D Gaussian Splatting model with the following command:
python g_train_all.py --dataset_name 3d_ovs --scene room --config_file config/w1_v5.yamlUse the interactive GUI to visualize 3D scene segmentation results (supports single/multi-object segmentation and occlusion region inspection). The GUI provides intuitive controls for view adjustment, segmentation operation, and rendering parameter tuning.
# General comman
python labelgs_gui.py -m {model_output_path} -s {scene_path}
# Example (replace placeholders with actual paths)
python labelgs_gui.py -m output/3d_ovs/w1_room_v5 -s dataset/3d_ovs/room| Operation/Control | Function Description |
|---|---|
| Mouse Left Drag | Hold and drag the left mouse button to rotate/translate the 3D scene view (adjust perspective freely). |
| Mouse Right + Segment3D Button | First click the segment3d button in the GUI, then hold the right mouse button to select target objects. Multiple objects can be selected in one operation for simultaneous segmentation. |
| Clear Button | Click to cancel all existing segmentation results and reset the selection state. |
| Playing Toggle | Check the playing option to enable automatic playback of rendering effects across different pre-defined perspectives (no manual view adjustment needed). |
| Reset_Playing Button | Click to stop the automatic playback and reset the view to the initial perspective. |
| Scale Slider | Drag the Scale slider to adjust the radius of 3D Gaussians (affects the visual granularity of the 3D scene). |
| Training FOVy Slider | Drag the training fovy slider to modify the vertical field of view (FOVy) of the virtual camera, adjusting the visible range of the 3D scene. |
Segmenting one object.
figuren1-1.mp4
Segmentint multiple objects simultaneously.
fingure3.mp4
Segmenting the occluded object and showing the occluded region.
fingure2-2.mp4
Evaluate the model’s performance on PSNR (reconstruction quality) and mIoU (segmentation accuracy) with:
# Evaluation script (uses the same config as training for consistency)
python eval/g_eval_render_all.py --dataset_name 3d_ovs --scene room --config_file config/w1_v5.yaml@inproceedings{zhang2025labelgs,
title={LabelGS: Label-Aware 3D Gaussian Splatting for 3D Scene Segmentation},
author={Yupeng Zhang, Dezhi Zheng, Ping Lu, Han Zhang, Lei Wang, Liping xiang, Cheng Luo, Kaijun Deng, Xiaowen Fu, Linlin Shen, Jinbao Wang},
journal={arXiv preprint arXiv:2508.19699},
year={2025}
}