The benchmark/ module runs the OpenSfM pipeline on a fixed set of datasets at a specific git commit, collects per-step timings and quality statistics, and produces an HTML report that compares one commit against another. It's the tool to answer "did my change make things faster / more accurate / break anything?".
Each benchmark run is fully isolated from your working tree:
- Resolve the commit you ask for (hash, branch, tag, or
HEAD). - Check it out in a throwaway git worktree under
.benchmark-worktrees/<hash>/— your current checkout is never touched. - Build it in a dedicated conda env (
opensfm-bench-<hash>) created from the commit's own lock file (conda-<os>-<arch>.lock, falling back toconda.yml), so the binary matches that commit exactly. - Set up each dataset as a lightweight directory (an
image_list.txtof absolute paths to the source images, ancillary files copied, and the chosen workflow config installed withprocessesset to the machine's CPU count). - Run the pipeline step by step, timing each one.
- Generate
comparison.html, diffing this run's metrics against a reference run. - Tear down the worktree and conda env.
Results land in benchmark_runs/<short-hash>_<timestamp>/. Both .benchmark-worktrees/ and benchmark_runs/ are git-ignored.
The orchestrator also hardens itself against the Linux OOM killer: it lowers its own OOM score and raises the pipeline subprocesses', so under memory pressure the kernel kills a pipeline step (recoverable with
--resume) rather than the whole benchmark.
- A git checkout of the repo (worktrees require it — a source tarball won't work) and
condaonPATH. - A datasets root: a directory with one subfolder per dataset, each containing an
images/folder. Optional per-dataset extras are picked up automatically:gcp_list.txt,ground_control_points.json,camera_models_overrides.json,exif_overrides.json, and amasks/folder.
A small JSON file describes what to run. See benchmark/benchmark_example.json:
{
"root": "./data",
"datasets": {
"berlin": "terrestrial",
"lund": "terrestrial"
},
"output_dir": "./benchmark_runs"
}| Field | Meaning |
|---|---|
root |
Directory containing the dataset subfolders. Must exist. |
datasets |
Map of dataset_name → workflow config. The config name resolves to configs/<name>.yaml (aerial, terrestrial, object). |
output_dir |
Where run directories are written (default ./benchmark_runs). |
The config is validated up front: the root, every dataset folder, every images/ subfolder, and every referenced configs/<name>.yaml must exist.
The module is invoked with python -m benchmark.run. The simplest run benchmarks a commit:
python -m benchmark.run --config benchmark/benchmark_example.json --commit HEADAdd --dense to also run the dense stages so their timings appear in the report:
python -m benchmark.run --config benchmark/benchmark_example.json --commit HEAD --denseUseful flags:
| Flag | Purpose |
|---|---|
--commit <ref> |
Commit/branch/tag to benchmark (starts a new run). Mutually exclusive with --resume. |
--resume <run_dir> |
Resume or re-run an existing run directory. |
--dense |
Also run undistort + the dense stages before compute_statistics. |
--from-step <step> |
Start at this step (see FAQ). |
--reference <run_dir | commit-prefix> |
What to compare against (default: most recent other run). |
--bootstrap <run_dir> |
With --commit --from-step, reuse earlier-step outputs from this run. |
--local-staging / --scratch-dir <dir> |
Process on a local disk and mirror back (for NAS run dirs). |
--report-only |
Regenerate the HTML only (requires --resume). |
--output-dir <dir> |
Override output_dir (ignored with --resume). |
-v / --verbose |
Debug logging. |
The typical A/B workflow is two runs — a baseline and your change:
# 1. Baseline (e.g. master)
python -m benchmark.run --config cfg.json --commit master
# 2. Your feature branch, compared explicitly against the baseline
python -m benchmark.run --config cfg.json --commit my-feature --reference master--reference accepts either a run directory or a commit-hash prefix (it searches output_dir for a matching run). If you omit it, the most recent other run is used automatically — so simply running two commits back-to-back already produces a comparison.
Open benchmark_runs/<run>/comparison.html. Each dataset shows a reference row and a current row across several metric groups, with cells coloured green (better) / red (worse):
- Reconstruction Summary — components, reconstructed shots/points, observations
- Reprojection Errors — normalized / pixel / angular
- Track Statistics — average track length (and length > 2)
- Feature Statistics — detected & reconstructed features (mean/median)
- GPS Errors / GCP Errors — average error
- Processing Times — per-step wall times (plus the dense stages when either run used
--dense), and total time
A dataset that failed shows a FAILED banner with the step it died on; the dataset header links to that dataset's stats/report.pdf. All numbers are read from each dataset's stats/stats.json (produced by compute_statistics); see quality report for what they mean.
benchmark_runs/
└── a1b2c3d4_20260628_124501/ # <short-hash>_<UTC-timestamp>
├── run_meta.json # commit, status, per-dataset per-step timings + success/stderr
├── comparison.html # the A/B report
├── berlin/ # a lightweight dataset dir (image_list.txt, config.yaml,
│ ├── ... # exif/, features/, reconstruction.json, reports/, stats/ …)
│ └── stats/{stats.json,report.pdf}
└── lund/
└── ...
run_meta.json is written after every dataset (and updated as steps complete), which is what makes a run crash-safe and resumable.
SfM (always): extract_metadata → detect_features → match_features → create_tracks → reconstruct → compute_statistics → export_report.
Dense (with --dense, spliced in before compute_statistics): undistort → dense_clustering → dense_equalize → compute_depthmaps → fuse_depthmaps → dense_merging (run with --georeferenced). See dense reconstruction.
Point --resume at the run directory. It reads the commit from run_meta.json (or infers it from the folder name), then skips every step whose output already exists and continues where it stopped:
python -m benchmark.run --config cfg.json --resume benchmark_runs/a1b2c3d4_20260628_124501Use --from-step with --resume. Steps before it are skipped unconditionally; that step and everything after it are re-run even if their outputs already exist (use this after changing, say, reconstruction code):
python -m benchmark.run --config cfg.json --resume <run_dir> --from-step reconstructValid steps: extract_metadata, detect_features, match_features, create_tracks, reconstruct, undistort, dense_clustering, dense_equalize, compute_depthmaps, fuse_depthmaps, dense_merging, compute_statistics, export_report.
--report-only (requires --resume) skips the worktree, build and pipeline entirely and just rebuilds comparison.html from existing results. Handy after the pipeline finished but you want a different comparison:
python -m benchmark.run --config cfg.json --resume <run_dir> --report-only --reference <other_run_or_prefix>SfM-only is the default — just omit --dense. Add --dense for the dense stages too. Selecting a dense --from-step (e.g. compute_depthmaps) implies --dense automatically. On --resume, the dense setting is inherited from the resumed run's run_meta.json unless you pass --dense again.
Combine --commit with --from-step: outputs of the steps before --from-step are symlinked/copied from an existing run. Without --bootstrap, the most recent complete run for the same commit is auto-detected:
# reuse everything up to reconstruct from a prior run of this commit
python -m benchmark.run --config cfg.json --commit my-feature --from-step reconstruct
# ...or name the source explicitly
python -m benchmark.run --config cfg.json --commit my-feature --from-step reconstruct --bootstrap <run_dir>Each dataset is independent: a failure stops that dataset at its failing step (recorded in run_meta.json as failed_step plus the tail of stderr) and the report shows a FAILED banner, but the remaining datasets still run. Re-run with --resume to retry just the unfinished steps, or --resume --from-step <step> to force a redo.
Use --local-staging (optionally with --scratch-dir <fast-local-dir>): each dataset is processed on local scratch and mirrored back to the run directory once, so network I/O doesn't pollute the timed steps.
The conda env is created from the benchmarked commit's conda-<os>-<arch>.lock (or conda.yml). Make sure that commit actually contains a valid lock file / conda.yml. Stale worktrees in .benchmark-worktrees/ are force-removed and recreated each run; you can safely delete that directory between runs.
- Workflow presets (
configs/) — the per-dataset configs the benchmark installs. - Quality report — the metrics surfaced in the comparison.
- Pipeline commands — the steps being benchmarked.