Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes with Selective VLM Reasoning

Juno Kim*, Yesol Park*, Hye-Jung Yoon*, Byoung-Tak Zhang

Seoul National University  •  IROS 2026

_{*Equal contribution}

Abstract

Indoor mobile robots require open-vocabulary scene understanding that grounds natural-language queries in a consistent 3D map. Many existing systems ultimately rely on cosine-similarity retrieval with contrastive image–text encoders, which is efficient but brittle when labels are near-synonymous or multiple similar instances appear. We present Scene-Q, a confidence-aware coarse-to-fine querying framework that normalizes encoder scores with temperature scaling and selectively invokes a reasoning VLM only for low-confidence cases. High-confidence queries are answered by fast retrieval, while ambiguous ones are reranked over a small top-K candidate set using the original multi-view images and instance bounding boxes, enabling context-aware disambiguation at low cost. Scene-Q improves open-vocabulary 3D instance segmentation on ScanNet200 and natural-language 3D instance retrieval on real-world reconstructions, with the largest gains on spatial and relational queries while keeping a substantial fraction of queries on the fast path.

Method

Scene-Q builds a queryable 3D instance map in three stages:

1. 3D instance segmentation via 2D-guided refinement. Starting from class-agnostic 3D masks (Mask3D / OneFormer3D) and per-view 2D masks (SAM / CropFormer), a superpoint-based, 2D-guided refinement step detaches over-merged components and merges neighboring instances that are consistent across views, yielding a clean refined instance set.
2. Multi-view instance descriptors. For each instance we select the top-k most-visible views, crop the object at multiple expansion levels, encode the crops with an image–text encoder (CLIP or SigLIP 2), and average them into a single ℓ2-normalized instance embedding.
3. Confidence-aware selective VLM reasoning. Cosine scores against the label set are passed through a temperature-scaled softmax; confidence statistics (p_max, top-1/top-2 margin, normalized entropy) route each query. High-confidence queries take the encoder's top-1 (fast path, ~50 ms), while low-confidence queries send the top-K labels plus the original multi-view images with bounding boxes to a reasoning VLM (Qwen2.5-VL) that picks the final label from full-image context.

Results

Open-vocabulary 3D instance segmentation on ScanNet200 (validation, 200 categories):

Method	mAP	AP@50	AP@25	head	common	tail
OVIR-3D	13.0	24.9	32.3	14.4	12.7	11.7
Open3DIS	23.7	29.4	32.8	27.8	21.2	21.8
OpenYOLO3D	24.5	31.7	36.2	27.8	24.3	21.6
Scene-Q (Ours)	25.2	33.7	41.6	25.9	24.2	25.7

Scene-Q achieves the best open-vocabulary results, with its largest gain under relaxed IoU (+5.4 AP@25 over OpenYOLO3D) and a balanced head/tail profile.

Ablation — selective VLM reasoning on ScanNet200 with oracle masks (mAP):

Encoder	None	Selective	Always
CLIP	32.9	43.6 (+32.5%)	42.8
SigLIP 2	48.4	51.9 (+7.2%)	50.8

Selective routing beats both never- and always-calling the VLM, confirming that reasoning is most valuable precisely on the ambiguous subset.

See the paper for full tables, including real-world natural-language retrieval (Table III).

Installation

Scene-Q uses two uv-managed virtual environments (no conda required) — a torch-2.4 stack for the semantic + VLM stage and a torch-1.11 stack for the mask/3D stages. One-command install for the main pipeline:

bash scripts/setup_env.sh

Full, step-by-step instructions (including the optional mask/3D-stage environment) are in INSTALL.md.

Data & weights

Datasets, model checkpoints, and outputs live outside the repo (they are git-ignored and typically symlinked to a data volume). See docs/DATA.md for the expected data/, weights/, and output/ layout, and for how to build the oracle masks. The image–text encoders (CLIP, SigLIP 2) and the reasoning VLM (Qwen2.5-VL) are downloaded from the HuggingFace / OpenAI hubs at runtime.

Reproduce ScanNet200

All commands are run from the repository root.

# (optional) fit the temperature T on the held-out tuning split
bash scripts/fit_temperature.sh scannet200 oracle siglip2_so

# Run the full pipeline (instance descriptors + confidence-aware routing).
# Single GPU, single process:
bash scripts/SceneQ_sem.sh scannet200 oracle siglip2_so 0 0 1

# Or fan out across 4 GPUs (parts 0..3):
bash scripts/run_SceneQ_sem_multi_process.sh scannet200 oracle siglip2_so 4 spread

# Evaluate (AP / AP@50 / AP@25)
bash scripts/eval.sh scannet200 oracle siglip2_so

SceneQ_sem.sh takes CONFIG_TYPE MODEL_3D SEM_MODEL GPU PART SUBPROCESS_NUM. Choose MODEL_3D from {oracle, mask3d, oneformer3d} and SEM_MODEL from {clip, siglip2_so, siglip2_giant}. Behaviour is configured in configs/scannet200.yaml.

Repository layout

SceneQ/          Core package: run_sem_pred.py (main), run_inst_seg.py,
                 fit_temperature.py, semantic/ (encoders, Qwen2.5-VL router),
                 instance/ (2D-guided 3D refinement)
configs/         scannet200.yaml — the ScanNet200 experiment configuration
evaluation/      ScanNet200 evaluator (ScanNetEval) + dataset category defs
scripts/         Launchers: SceneQ_sem.sh, eval.sh, fit_temperature.sh, ...
utils/           Projection, geometry, and I/O utilities
libs/            Vendored deps: CropFormer, pointops, Qwen2_5_VL wrapper
docs/            Documentation & figures

Acknowledgements

Scene-Q builds on excellent prior work, including Segment Anything, CropFormer / EntitySeg, Qwen2.5-VL, OpenMask3D, and Open3DIS.

Citation

If you find Scene-Q useful, please cite:

@inproceedings{kim2026sceneq,
  title     = {Scene-Q: Confidence-Aware Coarse-to-Fine Querying of 3D Scenes
               with Selective VLM Reasoning},
  author    = {Kim, Juno and Park, Yesol and Yoon, Hye-Jung and Zhang, Byoung-Tak},
  booktitle = {2026 IEEE/RSJ International Conference on Intelligent Robots and
               Systems (IROS)},
  year      = {2026}
}

License

Released under the MIT License.