Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 12 additions & 43 deletions docs/features/depth_estimation.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
**Category:** Lifting (2D → 3D)
**Experimental:** No

Depth estimation predicts the per-pixel distance from the camera for every pixel in a 2D RGB image, producing a depth map and optionally unprojecting it into a 3D point cloud or surface mesh. vizion3d uses [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) as its default backend.
Depth estimation predicts the per-pixel distance from the camera for every pixel in a 2D RGB image, producing a depth map and optionally unprojecting it into a 3D point cloud. vizion3d uses [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) as its default backend.

---

Expand Down Expand Up @@ -40,7 +40,6 @@ Set `VIZION3D_MODEL_CACHE` in your environment to change the default cache direc
| `model_backend` | `str` | No | vizion3D release checkpoint URL | Model backend identifier. See [Model backends](#model-backends) above. |
| `return_depth_image` | `bool` | No | `False` | If `True`, the result includes a 16-bit grayscale Open3D Image of the depth map. |
| `return_point_cloud` | `bool` | No | `False` | If `True`, the result includes an Open3D PointCloud unprojected from the RGB-D image. |
| `return_mesh` | `bool` | No | `False` | If `True`, the result includes an Open3D TriangleMesh reconstructed from the point cloud via ball-pivoting. |
| `advanced_config` | `DepthEstimationAdvanceConfig` | No | PrimeSense defaults | Camera intrinsics and depth range settings. See [Advanced config](#10-advanced-config-camera-intrinsics-depth-range) below. |

---
Expand All @@ -57,7 +56,6 @@ Set `VIZION3D_MODEL_CACHE` in your environment to change the default cache direc
| `backend_used` | `str` | Yes | Resolved model identifier that processed the request (local file path). |
| `depth_image` | `open3d.geometry.Image \| None` | When `return_depth_image=True` | 16-bit grayscale image, dtype `uint16`, shape `(H, W)`. The full 0–65535 range maps to `[min_depth, max_depth]`. |
| `point_cloud` | `open3d.geometry.PointCloud \| None` | When `return_point_cloud=True` | Coloured 3D point cloud unprojected from the RGB-D image using the intrinsics in `advanced_config`. Coordinates are in metres. |
| `mesh` | `open3d.geometry.TriangleMesh \| None` | When `return_mesh=True` | Triangle mesh surface reconstructed from the point cloud via ball-pivoting. Includes vertex colours. |
| `point_cloud_scale` | `float` | Yes | Scale factor: multiply any distance measured between two points in the point cloud by this value to get the equivalent distance in metres. Always `1.0` — Open3D produces point cloud coordinates directly in metres. |

---
Expand Down Expand Up @@ -154,33 +152,9 @@ o3d.io.write_point_cloud("scene.ply", pcd)

---

## 5. Surface mesh
## 5. All outputs at once

Request a triangulated mesh reconstructed from the point cloud via ball-pivoting. Includes vertex colours.

```python
import open3d as o3d
from vizion3d.lifting import DepthEstimation, DepthEstimationCommand

cmd = DepthEstimationCommand(
image_input="scene.png",
return_mesh=True,
)
result = DepthEstimation().run(cmd)

mesh = result.mesh # open3d.geometry.TriangleMesh
print(f"Vertices : {len(mesh.vertices)}")
print(f"Triangles : {len(mesh.triangles)}")

# Save as PLY
o3d.io.write_triangle_mesh("scene_mesh.ply", mesh)
```

---

## 6. All outputs at once

All three optional outputs can be requested in a single inference pass.
Both optional outputs can be requested in a single inference pass.

```python
import numpy as np
Expand All @@ -191,7 +165,6 @@ cmd = DepthEstimationCommand(
image_input="scene.png",
return_depth_image=True,
return_point_cloud=True,
return_mesh=True,
)
result = DepthEstimation().run(cmd)

Expand All @@ -204,14 +177,11 @@ depth_arr = np.asarray(result.depth_image) # uint16 (H, W)
# Point cloud
pcd = result.point_cloud
o3d.io.write_point_cloud("scene.ply", pcd)

# Mesh
o3d.io.write_triangle_mesh("scene_mesh.ply", result.mesh)
```

---

## 7. Custom model backend
## 6. Custom model backend

Use a local `.pth` checkpoint or a remote URL to a `.pth` file.

Expand Down Expand Up @@ -240,7 +210,7 @@ print(f"Backend: {result.backend_used}")

---

## 8. REST API
## 7. REST API

Start the server with all REST features enabled:

Expand Down Expand Up @@ -284,15 +254,14 @@ Send a request with `multipart/form-data`:
```bash
curl -X POST "http://localhost:8000/lifting/depth-estimation" \
-F "image=@scene.png" \
-F "return_point_cloud=true" \
-F "return_mesh=true"
-F "return_point_cloud=true"
```

The response is a JSON-serialised `DepthEstimationResult`. Binary fields (`depth_image`, `point_cloud`, `mesh`) are base64-encoded in the JSON response.
The response is a JSON-serialised `DepthEstimationResult`. Binary fields (`depth_image`, `point_cloud_ply`) are base64-encoded in the JSON response.

---

## 9. gRPC API
## 8. gRPC API

Start the server:

Expand Down Expand Up @@ -321,7 +290,6 @@ with open("scene.png", "rb") as f:
request = lifting_pb2.DepthEstimationRequest(
image_bytes=img_bytes,
return_point_cloud=True,
return_mesh=True,
)

response = stub.RunDepthEstimation(request)
Expand All @@ -332,7 +300,7 @@ print(f"Backend : {response.backend_used}")

---

## 10. Advanced config: camera intrinsics & depth range
## 9. Advanced config: camera intrinsics & depth range

`DepthEstimationAdvanceConfig` lets you supply the actual camera intrinsics and depth range for your sensor, replacing the built-in PrimeSense defaults. This is required for accurate metric 3D geometry when your camera is not a 640×480 PrimeSense sensor.

Expand Down Expand Up @@ -362,8 +330,9 @@ The same config is available in the REST and gRPC entry points. See [Advanced Co

---

---

## Known limitations

- **Relative depth only** — the default monocular backend produces relative (inverse) depth, not metric depth. Point cloud distances are internally consistent but not calibrated to real-world scale without a known reference distance.
- **Ball-pivoting mesh quality** — the mesh reconstructor works best on dense, evenly sampled point clouds. Sparse or noisy clouds may produce gaps or missing faces.
- **Python 3.12 required for Open3D** — `return_depth_image`, `return_point_cloud`, and `return_mesh` require Open3D, which currently only supports Python 3.12 in this project.
- **Python 3.12 required for Open3D** — `return_depth_image` and `return_point_cloud` require Open3D, which currently only supports Python 3.12 in this project.
38 changes: 6 additions & 32 deletions docs/features/stereo_depth.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ The S2M2 architecture comes in four size variants. The correct one is detected
| `model_backend` | `str` | No | vizion3D release checkpoint URL | S2M2 checkpoint. See [Model backends](#model-backends) above. |
| `return_depth_image` | `bool` | No | `False` | If `True`, the result includes a 16-bit grayscale Open3D Image of the depth map. |
| `return_point_cloud` | `bool` | No | `False` | If `True`, the result includes an Open3D PointCloud in metres. |
| `return_mesh` | `bool` | No | `False` | If `True`, the result includes an Open3D TriangleMesh reconstructed via ball-pivoting. |
| `advanced_config` | `StereoDepthAdvancedConfig` | No | 1280×720 @ 100 mm baseline defaults | Camera intrinsics and inference settings. See [Advanced config](#advanced-config) below. |

---
Expand All @@ -74,7 +73,6 @@ The S2M2 architecture comes in four size variants. The correct one is detected
| `backend_used` | `str` | Yes | Resolved local file path of the checkpoint used. |
| `depth_image` | `open3d.geometry.Image \| None` | When `return_depth_image=True` | 16-bit grayscale image, dtype `uint16`. The full 0–65535 range maps to `[min_depth, max_depth]` in metres. |
| `point_cloud` | `open3d.geometry.PointCloud \| None` | When `return_point_cloud=True` | Coloured 3D point cloud, coordinates in **metres**. |
| `mesh` | `open3d.geometry.TriangleMesh \| None` | When `return_mesh=True` | Surface mesh from ball-pivoting. Includes vertex colours. |
| `point_cloud_scale` | `float` | Yes | Always `1.0` — stereo depth produces real metric coordinates. |

---
Expand Down Expand Up @@ -187,28 +185,7 @@ o3d.io.write_point_cloud("scene.ply", pcd)

---

## 6. Surface mesh

```python
import open3d as o3d
from vizion3d.stereo import StereoDepth, StereoDepthCommand

cmd = StereoDepthCommand(
left_image="left.png",
right_image="right.png",
return_mesh=True,
)
result = StereoDepth().run(cmd)

mesh = result.mesh
print(f"Vertices : {len(mesh.vertices)}")
print(f"Triangles : {len(mesh.triangles)}")
o3d.io.write_triangle_mesh("scene_mesh.ply", mesh)
```

---

## 7. All outputs at once
## 6. All outputs at once

```python
import numpy as np
Expand All @@ -220,19 +197,17 @@ cmd = StereoDepthCommand(
right_image="right.png",
return_depth_image=True,
return_point_cloud=True,
return_mesh=True,
)
result = StereoDepth().run(cmd)

print(f"Depth range : {result.min_depth:.2f} → {result.max_depth:.2f} m")
depth_arr = np.asarray(result.depth_image) # uint16 (H, W)
o3d.io.write_point_cloud("scene.ply", result.point_cloud)
o3d.io.write_triangle_mesh("scene_mesh.ply", result.mesh)
```

---

## 8. Speed vs quality: scale factor
## 7. Speed vs quality: scale factor

Use `scale_factor < 1.0` to downsample input before inference for faster results:

Expand All @@ -251,7 +226,7 @@ result = StereoDepth().run(cmd)

---

## 9. REST API
## 8. REST API

Start the server with all REST features enabled:

Expand Down Expand Up @@ -305,11 +280,11 @@ curl -X POST "http://localhost:8000/lifting/stereo-depth" \
-F "return_point_cloud=true"
```

The response is a JSON-serialised `StereoDepthResult`. Binary fields (`depth_image`, `point_cloud_ply`, `mesh_ply`) are base64-encoded.
The response is a JSON-serialised `StereoDepthResult`. Binary fields (`depth_image`, `point_cloud_ply`) are base64-encoded.

---

## 10. gRPC API
## 9. gRPC API

Start the server:

Expand Down Expand Up @@ -422,5 +397,4 @@ cfg = StereoDepthAdvancedConfig(

- **Rectified pairs required** — images must be stereo-rectified so corresponding points lie on the same horizontal scanline. Un-rectified pairs will produce incorrect results.
- **Metric scale depends on calibration** — an incorrect `baseline` or `focal_length` scales all depth values uniformly. Always use calibrated values for real applications.
- **Ball-pivoting mesh quality** — works best on dense, evenly sampled point clouds. Sparse or noisy clouds from occluded regions may produce gaps or missing faces.
- **Python 3.12 required for Open3D** — `return_depth_image`, `return_point_cloud`, and `return_mesh` require Open3D, which currently only supports Python 3.12 in this project.
- **Python 3.12 required for Open3D** — `return_depth_image` and `return_point_cloud` require Open3D, which currently only supports Python 3.12 in this project.
4 changes: 1 addition & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ For per-backend prerequisites, install commands, and platform notes, see the [Ha

## Quick start — depth estimation

Get a depth map, point cloud, and mesh from a single image in under 10 lines.
Get a depth map and point cloud from a single image in under 10 lines.

```python
import open3d as o3d
Expand All @@ -56,7 +56,6 @@ result = DepthEstimation().run(
DepthEstimationCommand(
image_input="scene.png",
return_point_cloud=True,
return_mesh=True,
)
)

Expand All @@ -65,7 +64,6 @@ print(f"Points : {len(result.point_cloud.points)}")
print(f"Scale : {result.point_cloud_scale} metre per unit")

o3d.io.write_point_cloud("scene.ply", result.point_cloud)
o3d.io.write_triangle_mesh("scene_mesh.ply", result.mesh)
```

---
Expand Down
Binary file modified tests/assets/indoor_scene.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 1 addition & 4 deletions tests/integration/test_stereo_depth_direct.py
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ def test_stereo_direct_advanced_config_scale_factor(
def test_stereo_direct_all_outputs_returned(
stereo_image_pair, stereo_advanced_config, local_stereo_model_path
):
"""depth_image, point_cloud, and mesh are all returned when requested."""
"""depth_image and point_cloud are returned when requested."""
StereoDepthHandler._stereo_models.clear()
left_bytes, right_bytes = stereo_image_pair

Expand All @@ -243,7 +243,6 @@ def test_stereo_direct_all_outputs_returned(
model_backend=local_stereo_model_path,
return_depth_image=True,
return_point_cloud=True,
return_mesh=True,
advanced_config=stereo_advanced_config,
)
)
Expand All @@ -254,5 +253,3 @@ def test_stereo_direct_all_outputs_returned(
assert result.point_cloud is not None
assert result.point_cloud.has_points()
assert result.point_cloud_scale == 1.0

assert result.mesh is not None
2 changes: 0 additions & 2 deletions tests/unit/test_depth_estimation_advanced_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,6 @@ def _post(self, extra_data=None):
result.backend_used = "/fake/model.pth"
result.depth_image = None
result.point_cloud = None
result.mesh = None

with patch("vizion3d.server.rest.depth_estimation.DepthEstimation") as mock_cls:
mock_cls.return_value.run.return_value = result
Expand Down Expand Up @@ -322,7 +321,6 @@ def _run(self, proto_cfg=None):
result.backend_used = "/fake/model.pth"
result.depth_image = None
result.point_cloud = None
result.mesh = None

with patch("vizion3d.server.grpc.server.DepthEstimation") as mock_cls:
mock_cls.return_value.run.return_value = result
Expand Down
15 changes: 0 additions & 15 deletions tests/unit/test_depth_estimation_facade.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ def test_depth_estimation_basic(dummy_image_bytes):
assert res.backend_used.endswith(DEFAULT_DEPTH_MODEL_FILENAME)
assert res.depth_image is None
assert res.point_cloud is None
assert res.mesh is None
assert res.point_cloud_scale == 1.0


Expand Down Expand Up @@ -92,20 +91,6 @@ def test_point_cloud_orientation_keeps_image_top_up():
assert np.all(points[:, 2] > 0)


def test_depth_estimation_returns_mesh(dummy_image_bytes):
res = DepthEstimation().run(
DepthEstimationCommand(image_input=dummy_image_bytes, return_mesh=True)
)

assert isinstance(res.mesh, o3d.geometry.TriangleMesh)
assert res.mesh.has_vertices()
assert res.mesh.has_triangles()
assert res.mesh.has_vertex_colors()
import numpy as np

assert len(np.asarray(res.mesh.triangles)) > 0


def test_depth_estimation_accepts_file_path(tmp_path):
img = Image.new("RGB", (50, 50), color="red")
img_path = tmp_path / "test.png"
Expand Down
4 changes: 0 additions & 4 deletions tests/unit/test_depth_estimation_grpc.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ def _fake_result(depth_map=None):
result.backend_used = "/fake/model.pth"
result.depth_image = None
result.point_cloud = None
result.mesh = None
return result


Expand Down Expand Up @@ -84,7 +83,6 @@ def test_grpc_optional_fields_empty_when_not_requested(servicer, mock_context, i
response = servicer.RunDepthEstimation(request, mock_context)
assert response.depth_image == b""
assert response.point_cloud_ply == b""
assert response.mesh_ply == b""


def test_grpc_uses_default_backend_when_model_backend_is_empty(servicer, mock_context, image_bytes):
Expand Down Expand Up @@ -112,12 +110,10 @@ def test_grpc_forwards_return_flags(servicer, mock_context, image_bytes):
image_bytes=image_bytes,
return_depth_image=True,
return_point_cloud=True,
return_mesh=True,
)
with patch("vizion3d.server.grpc.server.DepthEstimation") as mock_cls:
mock_cls.return_value.run.return_value = _fake_result()
servicer.RunDepthEstimation(request, mock_context)
called_cmd = mock_cls.return_value.run.call_args[0][0]
assert called_cmd.return_depth_image is True
assert called_cmd.return_point_cloud is True
assert called_cmd.return_mesh is True
2 changes: 0 additions & 2 deletions tests/unit/test_depth_estimation_stereo_depth_rest_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ def _fake_result(depth_map=None):
result.backend_used = "/fake/model.pth"
result.depth_image = None
result.point_cloud = None
result.mesh = None
return result


Expand Down Expand Up @@ -116,7 +115,6 @@ def test_depth_estimation_optional_outputs_null_by_default(image_file):
data = response.json()
assert data["depth_image"] is None
assert data["point_cloud_ply"] is None
assert data["mesh_ply"] is None


def test_depth_estimation_backend_used_is_returned(image_file):
Expand Down
Loading
Loading