A lightweight framework for turning real-world videos and images into 3D maps, Blender scenes, and physical simulations β bridging feedforward reconstruction, sparse mapping, robotics, and physics in one CPU-friendly workflow.
Current release: v0.4.2 on PyPI
- v0.4.2 (2026-06-02) β PyPI; same-frame same-class 3D bbox NMS (discrete + progressive playback); class labels in separate Blender
BboxLabelscollection; bbox wireframes via curve bevel (bbox_wire_radius); playback camera clip range fix; billboard label orientation fixes. - v0.4.1 (2026-06-02) β PyPI; GPU/CUDA feedforward fixes (
TRANSFORMERS_NO_TF, transformers>=4.52auto-upgrade for RF-DETR); Blender nativepointcloudexport robustness;run_export_blend.shfor NPZ β.blend; README feedforward examples (ratio sampling,--detection_seg). - v0.4.0 (2026-06-02) β PyPI; RF-DETR instance segmentation (
--detection_seg) with masked 3D bboxes and Blender occupancy voxels; voxel-diff--algo_3d_bboxwithout detection; nestedoutput/blendYAML andreconstruct_config.json;point_displaymodes (pointcloud|points|spheres); ground align shifts leveled floor to z > 0 in Blender Z-up;random_points_per_frameratio defaults and shell no longer forces 4000;pip install vibephysics[detection_seg]optional extra. - v0.3.7 (2026-05-31) β PyPI; feedforward ground align (frame-0 camera up, 1D Hough multi-floor β bottom floor, bumpy-depth tilt); fixed-size Blender camera frustums/trajectory; point cloud icosphere instancing in blend export;
SKILL.mdground-align docs. - v0.3.6 (2026-05-31) β PyPI; DVLT feedforward (
--method dvlt);.vibephysics/feedforward/weight caches; Plotly trajectory aligned with saved poses; feedforwardSKILL.mdfor agents; GPU dependency fixes. - v0.3.5 (2026-05-31) β PyPI; feedforward stage timing/RSS; compact NPZ defaults (
min_confidence, per-frame/global sampling); Plotly frame-balanced sampling; R3 Mac/MPS kill warning. - 2026-05-30 β R3 / R3-Long; unified
run_feedforward.sh+feedforward.yaml; opt-in--blend/--html/--frames. - 2026-05-29 β Map-Anything, VGG-TTT.
- 2026-05-28 β VGGT-Omega; LingBot-Map long video; Blender Z-up
predictions.npz. - 2026-05-27 β GLOMAP/COLMAP mapping viz; Plotly HTML export.
Conda + pip install vibephysics (latest: 0.4.2); optional feedforward backends.
Installation steps
# 1. Create environment
conda create -n vibephysics python=3.11
conda activate vibephysics
# 2. Install core package (includes COLMAP/GLOMAP mapping & Blender simulation)
pip install "vibephysics>=0.4.2"
# 3. (Optional) Install feedforward backends from GitHub
# Or skip these β run_feedforward.sh auto-installs on first run
pip install git+https://github.com/robbyant/lingbot-map.git
pip install git+https://github.com/facebookresearch/vggt-omega.git
pip install "mapanything @ git+https://github.com/facebookresearch/map-anything.git"Sparse point clouds and camera poses in GSplat-ready sparse/0/ layout. Built on pycolmap 4.0+ (GLOMAP global mapper is integrated into COLMAP).
Usage
Python API (video, image folder, or single image):
from vibephysics import mapping
# GLOMAP β fast global mapper (default)
mapping.glomap_pipeline("test_home.mp4", output_path="mapping_output/test_home_glomap", matcher="sequential")
# COLMAP β incremental mapper
mapping.colmap_pipeline("path/to/images")CLI (reads src/vibephysics/mapping/configs/sfm.yaml, saves animated visualize.blend by default):
./run_glomap.sh --input test_home.mp4 --output_path mapping_output/test_home_glomap
./run_glomap.sh --input test_home.mp4 --no-blend # sparse only
./run_glomap.sh --input test_home.mp4 --no-animate # static .blendPress Spacebar in Blender to play the camera path animation (same style as feedforward .blend files).
Set engine: glomap or engine: colmap in the YAML. Use matcher: sequential for videos.
Visualize separately (if you used --no-blend):
bash run_glomap_visual.sh --sparse mapping_output/test_home_glomap/sparse/0 --output result.blendmapping.load_colmap_reconstruction("mapping_output/test_home_glomap/sparse/0", point_size=0.03, rotation=(-90, 0, 0))Output: sparse/0/ plus visualize.blend (unless --no-blend).
Feedforward 3D reconstruction from video or images via LingBot-Map, VGGT-Omega, VGG-TTT, Map-Anything, R3/R3-Long, and DVLT. By default, predictions.npz stores a compact colored point cloud plus camera poses; dense depth/world-point tensors are opt-in.
v0.4 highlights
- Blender point display:
output.blend.point_display: points(default) uses mesh vertices + geometry nodes. Setpointcloudfor native Blender point clouds (faster on very long sequences). Tune size withpoint_scale: 0.0035(default). - Adaptive sampling:
random_points_per_frame: 0.35keeps ~35% of confidence-filtered points per frame (float = ratio; scales with resolution and scene density). Prefer ratios over fixed counts like4000; use an integer only when you need an exact cap. - 2D β 3D object analysis:
--detection_segruns RF-DETR instance segmentation (COCO classes), then masked 3D wireframe bboxes, class labels, and semi-transparent occupancy voxels inscene.blend(see below).--algo_3d_bboxwithout detection = voxel-diff change blobs vs frame 0. - Optional:
pip install "vibephysics[detection_seg]"β otherwiserun_feedforward.sh/reconstructauto-install or upgradetransformers>=4.52on first--detection_seg.
RF-DETR per-instance masks β colored 3D bbox wireframes + billboard labels (BboxLabels collection). Toggle PointCloud, ChangeBBoxes, OccupancyVoxels, and BboxLabels in the outliner.
Feedforward setup & usage
Install backends (Python 3.11 + bpy). Pre-install from GitHub (see Installation) or let run_feedforward.sh auto-install on first use. Defaults: compact predictions.npz with ratio sampling (random_points_per_frame: 0.35); add --frames, --html, and --blend as needed. Blender export uses points display by default; set output.blend.point_display: pointcloud in YAML for faster native point clouds on huge scenes.
Examples below build up step by step on the same base command β each step adds one thing. Omitted flags use feedforward.yaml defaults.
pip install vibephysics bpy
# 1. Simplest β compact npz (default ~35% pts/frame after confidence filter)
./run_feedforward.sh --method lingbot_map --input test_recording.MOV
# 2. + preprocessed RGB frames folder
./run_feedforward.sh \
--method lingbot_map \
--input test_recording.MOV \
--frames
# 3. + Plotly browser viewer (uses frames/ for source-frame preview)
./run_feedforward.sh \
--method lingbot_map \
--input test_recording.MOV \
--frames \
--html
# 4. + Blender export (points display + point_scale 0.0035 by default)
./run_feedforward.sh \
--method lingbot_map \
--input test_recording.MOV \
--frames \
--blend
# 5. RF-DETR β masked 3D bboxes + class labels + occupancy voxels (assets/detection_seg_demo.png)
# Classes/colors: feedforward.yaml detection_seg.classes (COCO names, e.g. person, chair, couch)
# Blender layers: PointCloud | ChangeBBoxes | OccupancyVoxels | BboxLabels
./run_feedforward.sh \
--method lingbot_map \
--input test_recording.MOV \
--detection_seg \
--random_points_per_frame 0.35 \
--point_scale 0.0035 \
--frames \
--blend
# 6. Alternate engine β R3 on Mac/MPS (small batch; defaults otherwise)
./run_feedforward.sh \
--method r3 \
--input test_recording.MOV \
--max_frames 4 \
--frames
# 7. Map-Anything factory β e.g. Depth Anything 3
./run_feedforward.sh \
--method da3 \
--input path/to/images \
--blend
# 8. Full custom β output dir, frame limits, ratio caps, all exports
./run_feedforward.sh \
--method lingbot_map \
--input test_recording.MOV \
--output_path output/lingbot_map_demo \
--max_frames 24 \
--max_frames_mode first \
--random_points_per_frame 0.4 \
--detection_seg \
--frames \
--html \
--blendConfigs: src/vibephysics/feedforward/configs/
feedforward.yaml is the single feedforward config. It includes sections for all engines; run_feedforward.sh --method ... selects the active engine and patches runtime output flags.
Config (feedforward.yaml): one file for all engines. run_feedforward.sh sets engine from --method and patches output.*, output.blend.*, detection_seg.*, and algo_3d_bbox.* from CLI flags (--blend, --detection_seg, --point_scale, --random_points_per_frame, β¦). For R3, --method r3 / r3_long also sets r3.model.
engine: lingbot_map # lingbot_map | vggt_omega | vgg_ttt | map_anything | r3 | dvlt
image_path: path/to/images
output_path: null
verbose: true
video:
fps: 2 # extraction rate; cached in .vibephysics_extract_fps
quality: 2
max_frames: null # null = all frames; N limits count
max_frames_mode: first # first | spread
output:
save_blend: null # scene.blend path, or set by --blend
save_html: null
save_frames: false
min_confidence: 2.0
random_points_per_frame: 0.35 # float in (0,1] = ratio; int = max pts/frame; 0 = dense
total_random_points: 0 # float = global ratio cap; int = global max; 0 = off
align_ground: true
algo_3d_bbox: false # auto true when detection_seg.enabled
blend: # Blender-only (when save_blend set)
point_scale: 0.0035
point_display: points # points (default) | pointcloud (fast native) | spheres
animate: true
animation_fps: 24
animation_mode: progressive # progressive | discrete
detection_seg:
enabled: false # --detection_seg
model: Roboflow/rf-detr-seg-medium
classes: [person, cyan, chair, red, ...] # COCO names; "name, color" per line
threshold: 0.25
algo_3d_bbox:
voxel_size: 0.02
min_changed_voxels: 12
# masked_cluster_aabb when detection_seg on; voxel_diff_blob with --algo_3d_bbox alone
lingbot_map:
model: lingbot-map
checkpoint: null
image_size: 518
mode: auto # auto | streaming | batch
keyframe_interval: null
max_streaming_keyframes: null
window_size: 64
overlap_size: 16
overlap_keyframes: null
use_sdpa: false
mask_sky: false
vggt_omega:
checkpoint: null
checkpoint_name: vggt-omega-1b-512
resolution: 512
preprocess_mode: balanced
enable_alignment: false
conf_percentile: 50.0
depth_edge_rtol: 0.03
vgg_ttt:
model_id: nvidia/vgg-ttt
preprocess_mode: crop
image_size: 518
conf_percentile: 50.0
depth_edge_rtol: 0.03
num_ttt_steps: 1
memory_efficient_inference: false
map_anything:
model: vggt # model_factory key; see table below
model_kwargs: null
install_all: false
resolution: 518
norm_type: identity # vggt/pi3/moge=identity; mapanything/da3=dinov2; dust3r=dust3r
patch_size: 14
resize_mode: fixed_mapping # fixed_mapping | longest_side | square | fixed_size
size: null # required for longest_side / square / fixed_size
r3:
checkpoint: null # null = auto-download KevinXu02/R3
model: r3_long # r3 | r3_long (--method r3_long sets this)
config_name: r3-large
mode: local # test | local | long | strided
image_size: 504
kv_backend: dense # dense | paged (paged needs flashinfer)
rel_pose_method: greedy # greedy | pgo
metric_model_name: depth-anything/DA3METRIC-LARGEInput: folder, single image, or video (.mov/.mp4). Videos extract frames at video.fps into output/<video_stem>/ and reuse cached frames on reruns.
run_feedforward.sh routes direct engines (lingbot_map, vggt_omega, vgg_ttt, r3, r3_long, dvlt) and Map-Anything factory model keys (da3, mapanything, vggt, mast3r, pi3, etc.) through one CLI. Unknown method names are treated as Map-Anything model keys so new factory methods can be tried without changing the script.
Saved output defaults: predictions.npz is compact by default: min_confidence: 2.0 first, then random_points_per_frame: 0.35 keeps a ratio of surviving points per frame (scales with input resolution β no fixed β4000 pointsβ default). Optional total_random_points as a float applies a second global ratio cap. Set --random_points_per_frame 0 for dense legacy arrays (depth, conf, world_points, β¦). Pass --blend for scene.blend (points display by default), --html for visual.html, --frames for RGB frames, --detection_seg for masks + 3D bboxes + voxels (see layout below).
Map-Anything model keys:
run_feedforward.sh --method <map-anything-key> uses the Map-Anything unified loader and converts outputs into the same FeedforwardPrediction format as LingBot-Map and VGGT-Omega.
| Model key | Default preprocessing | Notes |
|---|---|---|
mapanything |
resolution: 518, norm_type: dinov2, patch_size: 14 |
Official facebook/map-anything checkpoint via MapAnything.from_pretrained() |
mapanything_apache |
518, dinov2, 14 |
Apache-licensed facebook/map-anything-apache checkpoint |
mapanything_ablations |
518, dinov2, 14 |
Map-Anything ablation model key when available in the installed package |
vggt |
518, identity, 14 |
Default VibePhysics Map-Anything backend |
moge |
518, identity, 14 |
MoGe wrapper defaults to Ruicheng/moge-vitl |
pi3 |
518, identity, 14 |
Pi3 wrapper |
pi3x |
518, identity, 14 |
Pi3x wrapper; auto-installs the pi3 extra when needed |
dust3r |
512, dust3r, 16 |
Downloads the official DUSt3R checkpoint if no ckpt_path is supplied |
mast3r |
512, dust3r, 16 |
Downloads the official MASt3R checkpoint if no ckpt_path is supplied |
must3r |
512, dust3r, 16 |
Downloads official MUSt3R checkpoints if paths are not supplied |
modular_dust3r |
512, dust3r, 16 |
Modular DUSt3R key when available in the installed package |
pow3r |
512, dust3r, 16 |
Requires model_kwargs.ckpt_path for the Pow3R checkpoint |
pow3r_ba |
512, dust3r, 16 |
Pow3R with bundle adjustment; requires model_kwargs.ckpt_path |
anycalib |
518, dinov2, 14 |
AnyCalib wrapper; auto-installs the anycalib extra when needed |
da3 |
504, dinov2, 14 |
Depth Anything 3 wrapper; auto-installs depth-anything-3 extra when needed |
For model-specific arguments, set map_anything.model_kwargs in YAML. The run script auto-installs the selected model extra with numpy<2 pinned for bpy compatibility; use --install-all to install all Map-Anything extras or --no-install / VIBEPHYSICS_NO_AUTO_INSTALL=1 if you manage dependencies manually.
See Time-sync comparison for side-by-side .blend export (e.g. GLOMAP vs LingBot-Map).
Python API:
from vibephysics import feedforward
output_dir = feedforward.reconstruct_from_config(
"src/vibephysics/feedforward/configs/feedforward.yaml",
image_path="test_recording.MOV",
)
pred = feedforward.load_prediction(output_dir / "predictions.npz")
map_output_dir = feedforward.reconstruct_from_config(
"src/vibephysics/feedforward/configs/feedforward.yaml",
image_path="test_recording.MOV",
map_anything_model="vggt",
)| Engine | Best for | Frames |
|---|---|---|
| LingBot-Map | Long video, streaming | 100β25,000+ |
| VGGT-Omega | High-quality batches | 10β100 |
| VGG-TTT | Test-time training experiments | Small batches |
| Map-Anything | Trying many feedforward models behind one interface | Model-dependent |
| R3 / R3-Long | Online/streaming relative-pose reconstruction | Long videos; use small --max_frames on Mac/MPS |
Output layout:
feedforward_output/{engine}_{timestamp}/
predictions.npz # compact points+poses (ratio sampling by default)
reconstruct_config.json # nested output + blend + detection_seg sections
frames/ # optional (--frames)
visual.html # optional (--html)
scene.blend # optional (--blend); points display by default
detection_seg/ # optional (--detection_seg)
masks/ # per-instance PNG masks when detected
summary.json
algo_3d_bbox.json # 3D bboxes + voxel_centers for Blender viz
predictions.npz uses Blender Z-up (metadata.world_coordinates: blender_z_up). Ground align (align_ground: true, default) runs in OpenCV space before Z-up save: frame-0 camera pose sets rough up, 1D Hough voting along that axis finds multiple floor heights, and the lowest floor below the camera is leveled (works on bumpy depth, not a flat-plane assumption). Metadata may include ground_align_floor_count and ground_align_floor_heights. Blender import does not re-align or re-axis-convert. Re-export a saved run to .blend without re-inference:
./run_export_blend.sh --predictions output/feedforward_output/lingbot_map_*/predictions.npzUses reconstruct_config.json beside the NPZ for blend settings. Post-process an existing .blend with run_postprocess_blend.sh --point_scale SIZE.
Compact predictions are best when you only need a colored 3D point cloud, trajectory, and camera poses; dense mode is best when you need full per-pixel depth/confidence/world-point maps.
Plotly HTML point cloud:
./run_feedforward.sh --method lingbot_map --input test_recording.MOV --html
python -m vibephysics.feedforward.export plotly \
--predictions output/feedforward_output/lingbot_map_20260528_144552/predictions.npz \
--output output/feedforward_output/lingbot_map_20260528_144552/pointcloud_plotly.html \
--trajectoryThe HTML viewer renders all valid points saved in predictions.npz; density is controlled by random_points_per_frame / total_random_points ratios (or integers for hard caps). For manual ad-hoc export, you can still pass --max-points to downsample a large existing prediction. It draws the camera trajectory as red dots connected by a red line and includes Play/Pause buttons (1x to 16x) plus a frame slider. Install Plotly if needed:
Blender performance tips: default point_display: points is compatible across Blender versions. For very large compact exports, set point_display: pointcloud in YAML (native point clouds, faster to open/scrub). Use spheres only when you need round points. Lower --random_points_per_frame ratio (e.g. 0.15) before lowering point_scale if the file is slow to open. --detection_seg adds bbox wireframes and voxel cubes per detected instance; tune algo_3d_bbox.min_visualize_changed_voxels in YAML to skip tiny blobs.
pip install plotlySide-by-side .blend with a shared timeline β scrub once, both reconstructions play in sync. Use the same input video and the same extraction fps (video.fps: 2 in both mapping and feedforward configs).
Compare workflow
1. Run both pipelines on the same input
./run_glomap.sh --input test_home.mp4 --output_path mapping_output/test_home_glomap
./run_feedforward.sh --method lingbot_map --input test_home.mp4 --output_path feedforward_output/lingbot_map_test_home2. Combine into one compare .blend
./run_compare_blend.sh \
--left mapping_output/test_home_glomap/sparse/0 \
--right feedforward_output/lingbot_map_test_home/predictions.npz \
--output compare_output/glomap_vs_lingbot.blendEach side can be:
predictions.npz(LingBot-Map, VGGT-Omega, VGG-TTT, Map-Anything, ...)sparse/0/folder from GLOMAP/COLMAP mapping
3. View in Blender
Open the compare .blend β split viewport (left vs right), shared timeline. Press Spacebar to play both animations together.
Feedforward vs feedforward works the same way:
./run_compare_blend.sh \
--left feedforward_output/vggt_omega_test/predictions.npz \
--right feedforward_output/lingbot_map_test/predictions.npz \
--output compare_output/vggt_vs_lingbot.blendPython API:
python -m vibephysics.feedforward.export compare \
--inputs mapping_output/test_home_glomap/sparse/0 \
feedforward_output/lingbot_map_test_home/predictions.npz \
--output compare_output/glomap_vs_lingbot.blend \
--video_fps 2Timing notes
- Both sides use the same animation model: duration β
(num_frames - 1) / video_fps - For a fair comparison, use the same video and same
video.fpswhen extracting frames - GLOMAP may register fewer cameras than extracted frames β its animation can be shorter than the source video
Robot walking with rigid body physics, uneven ground, puddles, and annotation overlay β sh run_robot.sh.
Run robot simulation
sh ./run_robot.sh
sh ./run_robot.sh mounted # POV (default)
sh ./run_robot.sh center # overview
sh ./run_robot.sh following # third-personBounding boxes, motion trails, and point cloud tracking β sh run_basics.sh.
Annotation demos
sh ./run_basics.shPer-point frustum culling; in-frustum points turn red in real time β sh run_basics.sh.
Frustum options
sh ./run_forest.sh --frustum-mode highlight
sh ./run_forest.sh --frustum-mode frustum_onlyBuoyancy, ripples, and point tracking β sh run_water.sh.
Water demo
sh ./run_water.shUnitree Go2 with water and debris β python examples/go2/go2_waypoint_walk.py.
Go2 commands
python examples/go2/go2_waypoint_walk.py
python examples/go2/go2_waypoint_walk.py --end-frame 150 --num-spheres 50CPU-friendly physics, robots, water, annotations, sparse mapping, and dense feedforward in one package.
Feature list
- π No GPU Required β Efficient on CPU-only machines; GPU optional for rendering.
- π€ Robot Simulation β IK walking with Open Duck and Unitree Go2.
- π§ Water Physics β Puddles, ripples, buoyancy.
- π Annotation Tools β Bboxes, motion trails, point tracking.
- πΊοΈ Sparse Mapping β GLOMAP global and COLMAP incremental SfM via pycolmap 4.0+.
- π§ Feedforward β LingBot-Map, VGGT-Omega, VGG-TTT, and Map-Anything.
- π§ Developer Friendly β Pure Python,
bpyas a module, no GUI required.
Python 3.11 + bpy; Blender 5.0 optional for viewing .blend files.
Details & third-party assets
- Python 3.11 (required for
bpy; 3.12+ not supported) - bpy (Blender as a Python module)
- Blender 5.0 β blender.org
β οΈ PyPIbpy5.0 ships cp311 wheels only.
- Open Duck: Open Duck Blender model
- Unitree Go2: Unitree model dataset
One-liner entry points for demos and simulations.
All quick-start commands
sh ./run_basics.sh
sh ./run_robot.sh
sh ./run_forest.sh
sh ./run_water.sh
python examples/go2/go2_waypoint_walk.py
./run_feedforward.sh --method lingbot_map --input test_recording.MOV
./run_feedforward.sh --method vggt --input test_recording.MOV
./run_glomap.sh --image_path path/to/imagesOpen output/*.blend in Blender 5.0 and press Spacebar to play.
Platform commands
open output/robot_waypoint.blend # macOS
blender output/robot_waypoint.blend # Linux
start output/robot_waypoint.blend # WindowsCenter, mounted, and following camera rigs; switch active camera in the Outliner.
Camera API & shell options
| Camera type | Description | Best for |
|---|---|---|
| Center | Circle around scene center | Overview |
| Mounted | On object (e.g. robot head) | POV |
| Following | Tracks target | Third-person |
from vibephysics.camera import CameraManager
cam_manager = CameraManager()
cam_manager.add_center_pointing('center', num_cameras=4, radius=25, height=12).create(target_location=(0, 0, 0))
cam_manager.add_object_mounted('mounted', num_cameras=4, distance=0.15).create(parent_object=robot_head, lens=10)
cam_manager.add_following('following', height=12, look_angle=60).create(target=robot_armature)
cam_manager.activate_rig('mounted', camera_index=0)sh run_robot.sh mounted | center | followingUse the green camera icon in the Outliner or Ctrl+Numpad 0 to switch cameras in Blender.
Import/export assets and initialize simulation scenes.
Setup API & formats
from vibephysics import setup
setup.init_simulation(start_frame=1, end_frame=250)
setup.load_asset('robot.glb')
setup.save_blend('output/scene.blend')| Import | Export |
|---|---|
| GLB/GLTF, FBX, PLY, OBJ, STL, DAE, USD, Blend | Blend, GLB, FBX, OBJ, PLY, STL, USD |
Viewer for 3D Gaussian splats (under development).
3DGS viewer
sh run_3dgs_viewer.shLicensed under the Apache License, Version 2.0.
License & citation
Copyright 2025 MIMI AI LTD
Licensed under the Apache License, Version 2.0. See LICENSE for the full text.
@misc{VibePhysics,
author = {Tsun-Yi Yang},
title = {VibePhysics: Physics and Robotics Simulation in Blender Without GPU Requirements},
month = {December},
year = {2025},
url = {https://github.com/mimiaigen/vibephysics}
}









