OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots

Juno Kim^1* · Yesol Park^1* · Hye-Jung Yoon^1* · Byoung-Tak Zhang^1,2,3

¹Interdisciplinary Program in AI, Seoul National University · ²AI Institute, SNU · ³Dept. of Computer Science, SNU
_{*Equal contribution}

IEEE/RSJ IROS 2024

       

TL;DR: OV-MAP builds zero-shot, open-vocabulary 3D instance maps for robots by lifting 2D class-agnostic masks into 3D with supplemented (raw + synthetic) depth and a 3D mask-voting mechanism — accurate per-instance 3D segmentation without any 3D-supervised network.

Abstract

We introduce OV-MAP, a novel approach to open-world 3D mapping for mobile robots by integrating open-features into 3D maps to enhance object recognition capabilities. A significant challenge arises when overlapping features from adjacent voxels reduce instance-level precision, as features spill over voxel boundaries, blending neighboring regions together. Our method overcomes this by employing a class-agnostic segmentation model to project 2D masks into 3D space, combined with a supplemented depth image created by merging raw and synthetic depth from point clouds. This approach, along with a 3D mask voting mechanism, enables accurate zero-shot 3D instance segmentation without relying on 3D supervised segmentation models. We assess the effectiveness of our method through comprehensive experiments on public datasets such as ScanNet200 and Replica, demonstrating superior zero-shot performance, robustness, and adaptability across diverse environments. Additionally, we conducted real-world experiments to demonstrate our method's adaptability and robustness when applied to diverse real-world environments.

Method

The pipeline (1) supplements raw depth with synthetic depth rendered from the reconstructed point cloud, (2) projects per-frame CropFormer 2D class-agnostic masks into 3D, (3) merges per-frame 3D masks and applies a dominant voting mechanism over mesh segments to obtain zero-shot 3D instances, and (4) labels each instance with CLIP features for open-vocabulary querying.

Contents

• Installation
• Data
• Usage
• Results
• Repository layout
• Citation
• Acknowledgements
• License

Installation

Setup uses uv with Python 3.10. See docs/INSTALL.md for the full step-by-step setup (PyTorch, detectron2 + CropFormer, pointops, CLIP, SAM). In short:

uv venv --python 3.10 && source .venv/bin/activate
uv pip install torch==1.11.0 torchvision==0.12.0 --index-url https://download.pytorch.org/whl/cu113
uv pip install -e .          # installs OV-MAP + pinned dependencies
# then follow docs/INSTALL.md for detectron2/CropFormer, pointops, CLIP, and SAM

Download the model checkpoints into weights/ as described in weights/README.md.

Data

OV-MAP is evaluated on ScanNet200. The expected data/ layout and the preprocessing steps are documented in docs/DATA.md. data/ and output/ are local directories (or symlinks) and are not tracked by git.

Usage

The pipeline runs from the repository root. End-to-end commands to reproduce the paper numbers are in docs/REPRODUCE.md.

# 1. Build the 3D instance map (CropFormer masks -> 3D merge -> dominant voting)
bash scripts/build_ovmap.sh cropformer sup 0        # [mask_model] [depth_type] [gpu]

# 2. Prepare mask input for evaluation
bash scripts/prepare_evaluation.sh

# 3. Compute CLIP mask features and evaluate (AP / AP50 / AP25)
bash scripts/run_scannet200_eval.sh cropformer sup 0

Equivalent module entry points (installed as console scripts): python -m ovmap.build_map (ovmap-build), python -m ovmap.prepare_evaluation (ovmap-prepare-eval), python -m ovmap.visualize (ovmap-visualize).

Results

ScanNet200 (validation, 3D instance segmentation)

Model	Open-Vocab	3D Proposal	Map Type	AP	AP50	AP25
Mask3D		Supervised	Per-Instance	26.9	36.2	41.4
OpenMask3D	✓	Mask3D	Per-Instance	15.4	19.9	23.1
OpenScene	✓	–	Per-Voxel	6.6	10.2	14.8
SAM3D	✓	None	Per-Instance	8.4	13.1	18.7
OV-MAP (ours)	✓	None	Per-Instance	11.9	17.4	23.2

3D Proposal = whether a (supervised) 3D segmentation network is used to generate proposals; Map Type = per-voxel vs per-instance representation.

Reproduced numbers may deviate slightly from the paper due to GPU type and inherent randomness in the pipeline.

Repository layout

ovmap/                 # the OV-MAP package (build_map, merging, depth_pc, mask_models, ...)
scripts/               # shell entry points for build / prepare / eval
open_query/            # CLIP mask-feature computation + evaluation (adapted OpenMask3D)
scannet-preprocess/    # ScanNet data preparation (adapted ScanNet/Pointcept)
libs/                  # vendored CropFormer, pointops (detectron2 installed separately)
weights/               # model checkpoints (downloaded, not committed)
docs/                  # INSTALL / DATA / REPRODUCE guides + figures

Vendored code (libs/, open_query/, scannet-preprocess/) is adapted from the upstream projects listed under Acknowledgements and retains their original licenses.

Citation

@inproceedings{kim2024ov,
  title={Ov-map: Open-vocabulary zero-shot 3d instance segmentation map for robots},
  author={Kim, Juno and Park, Yesol and Yoon, Hye-Jung and Zhang, Byoung-Tak},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  pages={13780--13786},
  year={2024},
  organization={IEEE}
}

Acknowledgements

OV-MAP builds on Segment Anything, Pointcept, SAM3D, CropFormer, and OpenMask3D. We thank the authors for releasing their code.

License

Released under the MIT License. Vendored third-party components remain under their respective licenses.