Skip to content

Add cross-library TSDF / ESDF / Occupancy benchmark drivers - #287

Draft
fwilliams wants to merge 1 commit into
mainfrom
feat/tsdf-esdf-benchmarks
Draft

Add cross-library TSDF / ESDF / Occupancy benchmark drivers#287
fwilliams wants to merge 1 commit into
mainfrom
feat/tsdf-esdf-benchmarks

Conversation

@fwilliams

Copy link
Copy Markdown
Collaborator

Summary

Adds a tests/benchmarks/tsdf_fusion/ benchmark suite that compares fvdb against the public sparse-voxel libraries (nvblox, VDBFusion, Open3D) on real and synthetic RGB-D / LiDAR workloads.

Pairs with the fvdb-core TSDF / ESDF / Occupancy / Decay PR (openvdb/fvdb-core#656, itself stacked on #655). The drivers exercise the new Grid.integrate_tsdf_frames, integrate_tsdf_from_points, integrate_occupancy_from_points, compute_esdf, and decay_and_prune APIs.

Per-dataset bench drivers (tests/benchmarks/tsdf_fusion/cross_library/)

Driver Dataset Workload
bench_open3d_vs_fvdb.py Synthetic sphere TSDF, fvdb vs Open3D
bench_replica_full.py Replica TSDF + ESDF, multi-scene, full frame rate
bench_esdf_replica.py Replica ESDF scale sweep
bench_kitti.py KITTI Odometry LiDAR TSDF
bench_esdf_kitti.py KITTI Odometry LiDAR ESDF
bench_occupancy_kitti.py KITTI Odometry LiDAR occupancy
bench_decay_kitti.py KITTI Odometry Dynamic-scene decay-and-prune
bench_mai_city.py Mai City (LiDAR) LiDAR TSDF, fvdb vs VDBFusion vs nvblox
bench_seven_scenes.py 7-Scenes (RGB-D) TSDF, long-trajectory
bench_esdf_vs_nvblox.py Mai City ESDF, fvdb vs nvblox
bench_occupancy_vs_nvblox.py Mai City Occupancy, fvdb vs nvblox

Supporting modules

  • kitti_loader.py, replica_loader.py, mai_city_loader.py, seven_scenes_loader.py — minimal per-dataset loaders that yield (intrinsics, cam_to_world, depth_or_points) tuples.
  • download_kitti.py, download_replica.py, download_replica_zip.py — resumable, parallel-stream downloaders for the public dataset archives.
  • nvblox_runner.py — Python wrapper around the nvblox CLI that matches the cross-library interface used by the bench drivers above.
  • install_nvblox.sh, install_vdbfusion.sh — reproducible install recipes for the two C++ comparison libraries (each builds against the fvdb conda env's TBB / Blosc / Boost).
  • data/README.md, data/.gitignore — conventions for placing dataset trees under data/. The dataset files themselves are not vendored.

Each driver accepts --help for its full argument surface and writes a JSON results file alongside per-frame stdout summaries.

Test plan

A no-external-dataset smoke test runs in a few seconds and is the easiest way to verify the setup:

python tests/benchmarks/tsdf_fusion/cross_library/bench_open3d_vs_fvdb.py \
    --n-frames 16 --json-out /tmp/smoke.json

The dataset-bound drivers expect the public archives under tests/benchmarks/tsdf_fusion/data/<dataset>/ (gitignored; downloaders provided in the same directory).

  • Synthetic sphere smoke test runs end-to-end with both fvdb and Open3D code paths and writes the JSON results file.
  • At least one dataset-bound driver runs end-to-end on its public archive (e.g. bench_replica_full.py against a Replica scene).
  • install_nvblox.sh / install_vdbfusion.sh produce working installs from a clean conda env (these are scripts, not part of the fvdb wheel).

Add a `tests/benchmarks/tsdf_fusion/` benchmark suite that compares
fvdb against the public sparse-voxel libraries (nvblox, VDBFusion,
Open3D) on real and synthetic RGB-D / LiDAR workloads. Pairs with the
fvdb-core `feat/tsdf-esdf-stack` PR.

What's included
---------------

Per-dataset bench drivers under `cross_library/`:

  bench_open3d_vs_fvdb.py      Synthetic sphere scene, TSDF only.
                               Self-contained smoke test (no external
                               dataset download).
  bench_replica_full.py        Replica multi-scene, TSDF + ESDF at
                               full frame rate.
  bench_esdf_replica.py        Replica ESDF scale sweep.
  bench_kitti.py               KITTI Odometry LiDAR TSDF.
  bench_esdf_kitti.py          KITTI Odometry LiDAR ESDF.
  bench_occupancy_kitti.py     KITTI Odometry LiDAR occupancy.
  bench_decay_kitti.py         KITTI Odometry dynamic-scene decay.
  bench_mai_city.py            Mai City LiDAR TSDF, fvdb vs VDBFusion
                               vs nvblox.
  bench_seven_scenes.py        7-Scenes long-trajectory TSDF.
  bench_esdf_vs_nvblox.py      Mai City ESDF, fvdb vs nvblox.
  bench_occupancy_vs_nvblox.py Mai City occupancy, fvdb vs nvblox.

Supporting modules:

  kitti_loader.py, replica_loader.py, mai_city_loader.py,
  seven_scenes_loader.py
      Minimal per-dataset loaders that yield
      `(intrinsics, cam_to_world, depth_or_points)` tuples.

  download_kitti.py, download_replica.py, download_replica_zip.py
      Resumable, parallel-stream downloaders for the public dataset
      archives.

  nvblox_runner.py
      Python wrapper around the nvblox CLI that matches the cross-
      library interface used by the bench drivers above.

  install_nvblox.sh, install_vdbfusion.sh
      Reproducible install recipes for the two C++ comparison
      libraries (each builds against the fvdb conda env's TBB /
      Blosc / Boost).

  data/README.md, data/.gitignore
      Conventions for placing dataset trees under `data/`; the dataset
      files themselves are not vendored.

How to use
----------

Each driver accepts `--help` for its full argument surface; typical
invocations are documented in the per-driver module docstrings. The
drivers write a JSON results file (path configurable via `--json-out`)
and print per-frame summaries to stdout.

`README.md` next to the suite indexes the drivers and lists the
supporting modules.

Signed-off-by: Francis Williams <francis@fwilliams.info>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant