VLM Spatial Benchmark

A benchmark for evaluating how well vision-language models (VLMs) reason about spatial safety on construction sites. The pipeline fuses 3D scene reconstruction (depth + point clouds) with object detection to produce grounded distance measurements, then compares grounded vs. ungrounded VLM safety assessments.

Overview

The core idea: given a construction site image, detect workers and machines, compute their real-world 3D distances, and ask a VLM whether any workers are at risk. Results from a grounded prompt (with metric distances) are compared against a baseline prompt (image only) using an LLM-as-judge scorer.

Repo Structure

vlm-spatial-benchmark/
├── pipeline/               # Core library
│   ├── blind_spot.py       # 3D proximity analysis — classifies workers as at-risk or safe
│   ├── combine.py          # Fuses SAM3 detections with MapAnything depth/point clouds
│   ├── crop.py             # Crops and masks individual detections from a full image
│   ├── describe.py         # Calls Claude to generate grounded safety descriptions
│   └── prompts.py          # Prompt templates (baseline, grounded, judge)
│
├── scripts/                # Entry points and utilities
│   ├── analyze_safety.py   # Run grounded or baseline safety analysis on an image
│   ├── score_results.py    # Score VLM outputs against ground truth using Claude as judge
│   ├── explore_data.py     # Explore and filter the dataset
│   ├── cross_ref.py        # Cross-reference results across runs
│   ├── print_results.py    # Pretty-print results JSON
│   └── visualize_segmentation.py  # Visualize SAM3 segmentation masks
│
├── notebooks/              # Colab notebooks for running the vision models
│   ├── SAM3.ipynb          # Text-prompted segmentation with SAM3
│   └── MapAnything.ipynb   # Metric depth estimation with MapAnything
│
├── data/                   # Local data (gitignored)
│   └── twelve_samples/     # Sample scene data (depth maps, point clouds, detections)
│
├── assets/                 # Visualizations and annotated outputs
└── pyproject.toml

Installation

Requires Python 3.12+.

git clone https://github.com/tantrik/vlm-spatial-benchmark.git
cd vlm-spatial-benchmark

Option 1 — uv (recommended)

# Install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

uv sync

Option 2 — pip

pip install -r requirements.txt

Set your API keys:

export ANTHROPIC_API_KEY=your_key_here
export GOOGLE_API_KEY=your_key_here   # for Gemini

Usage

Run safety analysis on an image

# Baseline (image only, no metric distances)
uv run python scripts/analyze_safety.py site.jpg

# Grounded (with 3D distance measurements)
uv run python scripts/analyze_safety.py site.jpg --provider claude

# Use Gemini instead
uv run python scripts/analyze_safety.py site.jpg --provider gemini

# Specify a model
uv run python scripts/analyze_safety.py site.jpg --provider claude --model claude-opus-4-6
uv run python scripts/analyze_safety.py site.jpg --provider gemini --model gemini-2.0-flash

Score results

# Score a single results file against ground truth
uv run python scripts/score_results.py --results_file path/to/results.json

# Compare grounded vs. baseline
uv run python scripts/score_results.py \
  --baseline path/to/baseline.json \
  --grounded path/to/grounded.json

Explore the dataset

The benchmark uses the LouisChen15/ConstructionSite dataset on HuggingFace.

uv run python scripts/explore_data.py

Visualize segmentation

uv run python scripts/visualize_segmentation.py

Notebooks

Two Colab notebooks cover the vision model steps of the pipeline:

Notebook	Description
SAM3.ipynb	Text-prompted instance segmentation — detects workers and excavators using SAM3
MapAnything.ipynb	Metric depth estimation — produces per-pixel depth maps and 3D point clouds using MapAnything

Both notebooks load images directly from HuggingFace and save results to .npz files consumed by pipeline/combine.py.

SAM3 Output	MapAnything Output

Dependencies

Package	Purpose
anthropic	Claude API (analysis + judge scoring)
google-genai	Gemini API
supervision	Bounding box annotation and visualization
scipy	Mask resizing, point cloud math
pillow	Image loading and cropping
datasets	HuggingFace dataset access
matplotlib	Plotting and visualization