3DCV Structure from Motion (SfM)

Structure from Motion on the box dataset: 51 photos of a small tin sitting on a marker board, taken while moving the camera around it. The pipeline detects and matches its own SIFT features, recovers the camera poses, and triangulates a 3D point cloud — then checks both against the ground truth.

Recovered camera path (red) and reconstructed point cloud.

Pipeline

1. Correspondences. Detect SIFT in every image and match all image pairs with a ratio test, a mutual nearest-neighbour check, and a fundamental-matrix RANSAC pass that keeps only the geometrically consistent inliers.
2. Incremental SfM. Bootstrap two views from the essential matrix, then add one camera at a time with PnP and triangulate new points. The tin is roughly symmetric, so the engine skips the camera that locks onto the mirror side, rejects pose jumps that break trajectory continuity, and guards each bundle adjustment against rotation flips and scale collapse.
3. Tracks. Link the pairwise matches into multi-view tracks — one 3D point per physical feature, observed in as many images as see it.
4. Global bundle adjustment. Optimise all cameras and track points together. The camera makes a half loop out and back, so the late views overlap the early ones; an L2 loss lets those long-range tracks pull the trajectory straight.

Results

Scored against the ground-truth poses and surface cloud, using the pipeline's own SIFT features. Camera 2 looks at the mirror-image side of the symmetric tin and is left out.

rot mean	rot median	rot max	<10°	<20°	reproj px	centre err	chamfer	points
1.64°	1.35°	2.87°	25/25	25/25	2.60	0.291	0.47	10,719

Every camera lands within 3° of ground truth.

Estimated (red) vs ground-truth (blue) cameras, the point cloud, and per-camera position error. The single tall bar is camera 2, the mirror view that is excluded from the metrics.

What makes this work

Two choices matter more than any parameter tuning. The bootstrap is far more accurate from the calibrated essential matrix than from a fundamental-matrix estimate, and the residual drift only disappears once the global bundle adjustment runs over multi-view tracks with an L2 loss — a robust loss treats the drift-correcting long-range matches as outliers and down-weights exactly the constraints that would straighten the trajectory.

stage	rot mean
incremental engine, sparse features	15.9°
incremental engine, tuned features	5.0°
+ multi-view tracks, robust loss	4.9°
+ multi-view tracks, L2 loss	1.6°

Feature density has a sweet spot — too few matches starves the late cameras, too many lets ambiguous matches across the symmetry slip through:

SIFT settings	pair files	rot mean (engine)
6k features, min 20 inliers/pair	653	15.9°
8k features, min 12 inliers/pair	1075	5.0°
12k features, min 10 inliers/pair	1266	13.1°

Usage

pip install -r requirements.txt
unzip data/box.zip -d data      # -> data/box/{images, *.json, gt_points.ply}

Run the whole thing and visualise it:

python run_pipeline.py                  # uses the cached features/sift if present
python run_pipeline.py --regen-features # rebuild the SIFT features first
python visualize_results.py --result runs/box

visualize_results.py writes an interactive sfm_3d.html and a sfm_results.png into the result folder, and opens a window unless you pass --no-show.

To compare the engine configurations behind the defaults:

python run_experiments.py --features features/sift

Use it on your own object

Nothing here is specific to the tin. Photograph any object from many angles — a slow full circle, well lit, with some surface texture — then:

1. drop the photos in data/box/images/;
2. put your camera's intrinsics in data/box/camera_parameters.json as {"intrinsics": [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]};
3. reconstruct and view it:

python run_pipeline.py --regen-features --max-images <N-1> --skip-cams ""
python visualize_results.py --result runs/box

--max-images is how many photos to use and --skip-cams "" keeps all of them (camera 2 is only skipped for this particular symmetric tin). Without ground-truth files the accuracy numbers are skipped, but you still get the camera path and the point cloud.

Layout

run_pipeline.py        end-to-end: features -> poses -> tracks -> cloud
run_experiments.py     sweep engine configurations, print a comparison table
visualize_results.py   estimated-vs-ground-truth dashboard (HTML + PNG)
eval_metrics.py        rotation / reprojection / Chamfer metrics
config.py              dataset paths and intrinsics
scripts/generate_sift.py   SIFT detection + matching
pipelines/robust_sfm.py    incremental pose estimation
pipelines/tracks.py        multi-view tracks + global bundle adjustment
core/                  fundamental matrix, triangulation, PnP, bundle adjustment

Notes

Only NumPy, SciPy and OpenCV do the geometry; Open3D and Plotly/Matplotlib are used for the Chamfer distance and the figures. The SIFT features and run outputs are regenerated from the images and are not committed; the dataset ships as data/box.zip.

Authors

Alejandro Campayo, Aishwarya Kshirsagar and Sai Suresh.