linting, checks and fixes

Author: OlafenwaMoses

docs/annotation/object_mask_annotation_3d.md

@@ -98,7 +98,15 @@ for ann in result.annotations:
 
 ## 2. Direct Python import — point cloud only (no image)
 
-Omit `image_input` and the task synthesises a front-view image from the point cloud itself.
+When `image_input` is omitted, the task synthesises a front-view RGB image directly from the point cloud's own XYZ+RGB data and runs segmentation on that synthetic view. This covers two common situations:
+
+**No image available at all** — the point cloud came from a file, a scan, or a pipeline that did not preserve the original photo. The synthesised view is the only option.
+
+**Stereo source with two images** — a stereo cloud is generated from a left and right image pair, but those are two separate images taken from slightly different viewpoints. There is no single image that naturally represents the combined stereo view. In this case, let the system synthesise the view from the point cloud — the synthesised view is computed from the cloud's 3D positions and stored colours, so it does not require choosing between the two frames. See [section 5](#5-stereo-point-cloud-integration) for the full stereo workflow.
+
+The synthesised image is a point-splatting projection: each point's XYZ is projected into pixel coordinates using the camera intrinsics, and its RGB colour is painted onto a canvas. For depth-estimation clouds (one point per pixel) the result is nearly identical to the original photo. For stereo clouds or scans with variable density, sparse or occluded regions produce a patchy image that may reduce detection quality compared to a real photo.
+
+> **Camera intrinsics required for non-PrimeSense clouds.** The default intrinsics (`fx=525, cx=319.5, cy=239.5`) match a 640×480 PrimeSense sensor. If your point cloud was generated by a different camera (stereo rig, RealSense, etc.), pass `advanced_config` with the correct values — otherwise back-projection will not align masks with the 3D points. See [Advanced config](#advanced-config).
 
 ```python
 import open3d as o3d
@@ -167,7 +175,91 @@ for i, ann in enumerate(result.annotations):
 
 ---
 
-## 5. REST API
+## 5. Stereo point cloud integration
+
+Point clouds produced by [Stereo Depth](../features/stereo_depth.md) are in camera space (X right, Y down, Z forward, origin at the left camera), which is exactly what this task expects. To annotate a stereo cloud correctly:
+
+- **Always pass the stereo camera intrinsics** via `advanced_config`. The default values are for a PrimeSense sensor and will not produce back-projection that matches any other stereo rig.
+- **Do not pass `image_input`** — a stereo cloud comes from two images taken at slightly different viewpoints and there is no single image that represents the combined view. Leave `image_input` unset and the system will synthesise the segmentation image directly from the point cloud's stored colours.
+- **Do not centroid-shift the point cloud** before passing it in. The PLY viewer handles visual centering in JavaScript; shifting the cloud in Python breaks the Z > 0 requirement that back-projection depends on.
+
+```python
+import open3d as o3d
+from vizion3d.annotation import ObjectMaskAnnotation3D, ObjectMaskAnnotation3DCommand
+from vizion3d.annotation.models import ObjectMaskAnnotation3DConfig
+
+pcd = o3d.io.read_point_cloud("stereo_result.ply")
+
+# Intrinsics must match the stereo rig used to generate the cloud.
+# Read these from your calib.txt: cam0=[fx 0 cx; 0 fy cy; 0 0 1]
+stereo_cfg = ObjectMaskAnnotation3DConfig(
+    fx=1733.74,
+    fy=1733.74,
+    cx=792.27,
+    cy=541.89,
+)
+
+result = ObjectMaskAnnotation3D().run(
+    ObjectMaskAnnotation3DCommand(
+        point_cloud=pcd,
+        return_annotated_cloud=True,
+        advanced_config=stereo_cfg,
+    )
+)
+
+for ann in result.annotations:
+    print(f"{ann.label:20s}  conf={ann.confidence:.2f}  3D points={len(ann.point_indices)}")
+
+o3d.io.write_point_cloud("annotated_stereo.ply", result.annotated_cloud)
+```
+
+The stereo pipeline can also generate the point cloud and annotate it in a single script:
+
+```python
+import open3d as o3d
+from vizion3d.stereo import StereoDepth, StereoDepthCommand, StereoDepthAdvancedConfig
+from vizion3d.annotation import ObjectMaskAnnotation3D, ObjectMaskAnnotation3DCommand
+from vizion3d.annotation.models import ObjectMaskAnnotation3DConfig
+
+# Step 1 — stereo depth → point cloud
+stereo_result = StereoDepth().run(
+    StereoDepthCommand(
+        left_image="left.png",
+        right_image="right.png",
+        return_point_cloud=True,
+        advanced_config=StereoDepthAdvancedConfig(
+            focal_length=1733.74,
+            cx=792.27,
+            cy=541.89,
+            baseline=536.62,
+        ),
+    )
+)
+
+# Step 2 — annotate the stereo cloud (reuse the same intrinsics)
+# image_input is omitted — the system synthesises the segmentation view from the cloud.
+annotation_result = ObjectMaskAnnotation3D().run(
+    ObjectMaskAnnotation3DCommand(
+        point_cloud=stereo_result.point_cloud,
+        return_annotated_cloud=True,
+        advanced_config=ObjectMaskAnnotation3DConfig(
+            fx=1733.74,
+            fy=1733.74,
+            cx=792.27,
+            cy=541.89,
+        ),
+    )
+)
+
+for ann in annotation_result.annotations:
+    print(f"{ann.label:20s}  conf={ann.confidence:.2f}  3D points={len(ann.point_indices)}")
+
+o3d.io.write_point_cloud("annotated_stereo.ply", annotation_result.annotated_cloud)
+```
+
+---
+
+## 6. REST API
 
 Start the server:
 
@@ -185,7 +277,7 @@ To preload the annotation checkpoint at startup:
 
 ```bash
 uv run vizion3d-serve-rest --object_mask_annotation_3d \
-  --annotation_model /models/yolo11n-seg.pt
+  --annotation_model /models/yolo26l-seg.pt
 ```
 
 Send a request with `multipart/form-data`. The `image` field is optional — omit it to let the server synthesise the front view.
@@ -208,7 +300,7 @@ curl -X POST "http://localhost:8000/annotation/object-mask-annotation-3d" \
 
 ```json
 {
-  "backend_used": "/path/to/yolo11n-seg.pt",
+  "backend_used": "/path/to/yolo26l-seg.pt",
   "annotations": [
     {
       "label": "chair",
@@ -226,7 +318,7 @@ curl -X POST "http://localhost:8000/annotation/object-mask-annotation-3d" \
 
 ---
 
-## 6. gRPC API
+## 7. gRPC API
 
 Start the server:
 

docs/concepts/camera_intrinsics.md

@@ -159,7 +159,7 @@ K  =    |   0.0  525.0   239.5 |
         |   0.0    0.0     1.0 |
 ```
 
-For a different camera or resolution, always supply calibrated values — wrong intrinsics produce correct topology but geometrically distorted metric scale.
+For a different camera or resolution, always supply calibrated values — intrinsics that do not match your camera produce correct topology but geometrically distorted metric scale.
 
 See the full field reference and per-entry-point usage examples in the Advanced Config pages:
 

docs/features/depth_estimation.md

@@ -381,8 +381,6 @@ The same config is available in the REST and gRPC entry points. See [Advanced Co
 
 ---
 
----
-
 ## Known limitations
 
 - **Relative depth only** — the default monocular backend produces relative (inverse) depth, not metric depth. Point cloud distances are internally consistent but not calibrated to real-world scale without a known reference distance.

docs/features/depth_estimation_advanced_config.md

@@ -16,7 +16,7 @@ X = (u - cx) * d / fx
 Y = (v - cy) * d / fy
 ```
 
-All four intrinsic parameters — `fx`, `fy`, `cx`, `cy` — appear in this formula. Getting them wrong produces a point cloud that is geometrically distorted: correct topology but wrong angles, skewed shapes, or objects that appear compressed or stretched.
+All four intrinsic parameters — `fx`, `fy`, `cx`, `cy` — appear in this formula. Values that do not match your camera produce a point cloud that is geometrically distorted: correct topology but skewed angles, compressed shapes, or stretched geometry.
 
 ---
 
@@ -44,7 +44,7 @@ fx = (image_width / 2) / tan(FoV_h / 2)
 
 The vertical focal length in pixels. For cameras with square pixels, `fy ≈ fx`. Cameras with non-square sensors may have `fy ≠ fx`.
 
-**Effect on the point cloud:** Controls vertical spread analogously to `fx`. Incorrect `fy` produces vertically compressed or stretched geometry.
+**Effect on the point cloud:** Controls vertical spread analogously to `fx`. A `fy` that does not match your sensor produces vertically compressed or stretched geometry.
 
 **How to find it:** `K[1][1]` from the calibration matrix, or:
 
@@ -60,7 +60,7 @@ fy = (image_height / 2) / tan(FoV_v / 2)
 
 The horizontal image coordinate of the optical axis — ideally the exact centre of the sensor. For a 640-wide image the ideal value is `319.5`; for a 1920-wide image it is typically near `959.5`.
 
-**Effect on the point cloud:** Shifts the entire point cloud left or right. A wrong `cx` makes the scene appear to be viewed from an off-centre vantage point, introducing a lateral tilt.
+**Effect on the point cloud:** Shifts the entire point cloud left or right. A `cx` that does not match your sensor makes the scene appear viewed from an off-centre vantage point, introducing a lateral tilt.
 
 ---
 
@@ -70,7 +70,7 @@ The horizontal image coordinate of the optical axis — ideally the exact centre
 
 The vertical image coordinate of the optical axis. For a 480-tall image the ideal value is `239.5`.
 
-**Effect on the point cloud:** Shifts the entire point cloud up or down. Like `cx`, an incorrect value introduces a tilt — vertical in this case.
+**Effect on the point cloud:** Shifts the entire point cloud up or down. Like `cx`, a value that does not match your sensor introduces a tilt — vertical in this case.
 
 ---
 

docs/features/depth_estimation_vs_stereo_depth.md

@@ -28,13 +28,19 @@ Provided the camera calibration is accurate, the output is **real metric depth i
 |---|---|---|
 | **Input** | Rectified left + right image pair | Single RGB image |
 | **Depth type** | Metric (real metres) | Relative (inverse depth, arbitrary scale) |
-| **Coordinate system** | Camera space | Camera space |
-| **Units** | Metres (real) | Metres (fictitious — mapped to `[0, depth_trunc]`) |
+| **Coordinate system** | Camera space (X right, Y down, Z forward) | Camera space (X right, Y down, Z forward) |
+| **Z ordering** | Near objects have smaller Z | Near objects have larger Z — relative depth only, not physical ordering |
+| **Units** | Metres (real) | Metres (relative, mapped to `[0, depth_trunc]`) |
 | **Object at 2.4 m reads as 2.4 m** | Yes — if calibration is correct | No — depends on scene content |
-| **Scale factor to world** | 1.0 (accurate) | Unknown, scene-dependent |
-| **`point_cloud_scale` field** | 1.0 (accurate) | 1.0 (misleading — not real metres) |
+| **Scale factor to world** | 1.0 (real) | Unknown, scene-dependent |
+| **`point_cloud_scale` field** | 1.0 (real metres) | 1.0 (relative, not real metres) |
 | **Shape / topology correct** | Yes | Yes, if correct intrinsics supplied via `DepthEstimationAdvanceConfig` |
 | **Camera calibration needed** | Yes — `focal_length`, `baseline`, `cx`, `cy` | Optional — only affects point cloud geometry |
+| **Compatible with annotation task** | ✅ | ✅ back-projection is self-consistent |
+| **Compatible with other 3D tools** | ✅ registration, reconstruction, metric tools | ⚠️ Z ordering is relative — not directly interoperable with metric clouds |
+| **Output format** | Open3D `PointCloud` | Open3D `PointCloud` |
+| **PLY export** | ✅ | ✅ |
+| **MPS inference** | float32 | float32 |
 | **Input requirements** | Stereo rig, rectified images | Any single photo |
 | **Depth completeness** | Gaps in occluded / textureless regions | Dense — every pixel has a prediction |
 | **Runtime** | Moderate (transformer-based matching) | Moderate (ViT-based encoder-decoder) |
@@ -78,7 +84,7 @@ result = DepthEstimation().run(
 points = np.asarray(result.point_cloud.points)  # shape (N, 3)
 # point_cloud_scale == 1.0, but distances are NOT real metres —
 # the depth model output is relative and mapped to depth_trunc.
-print(f"point_cloud_scale: {result.point_cloud_scale}")  # 1.0 (misleading)
+print(f"point_cloud_scale: {result.point_cloud_scale}")  # 1.0 (relative, not real metres)
 ```
 
 ### Stereo point cloud (Stereo Depth)
@@ -113,10 +119,10 @@ print(f"point_cloud_scale: {result.point_cloud_scale}")   # 1.0 (accurate)
 
 | Output field | Depth Estimation | Stereo Depth |
 |---|---|---|
-| `depth_map` | Relative depth (fictitious metres) | Metric depth (real metres) |
+| `depth_map` | Relative depth (not real metres) | Metric depth (real metres) |
 | `disparity_map` | Not present | Pixel disparity (always returned) |
 | `min_depth` / `max_depth` | Relative range | Real range in metres |
-| `point_cloud_scale` | 1.0 (misleading) | 1.0 (accurate) |
+| `point_cloud_scale` | 1.0 (relative, not real metres) | 1.0 (real metres) |
 | `backend_used` | Local path to Depth Anything V2 `.pth` | Local path to S2M2 `.pth` |
 
 ---

docs/features/stereo_depth.md

@@ -468,8 +468,55 @@ cfg = StereoDepthAdvancedConfig(
 
 ---
 
+## 3D annotation from a stereo cloud
+
+A stereo point cloud is in camera space (Z = metric depth, origin at the left camera), making it directly compatible with [Object Mask Annotation 3D](../annotation/object_mask_annotation_3d.md). Pass the same intrinsics you used for stereo depth. Do not pass `image_input` — the annotation task synthesises the segmentation image from the point cloud's stored colours, which avoids having to pick between the left and right frames.
+
+```python
+import open3d as o3d
+from vizion3d.stereo import StereoDepth, StereoDepthCommand, StereoDepthAdvancedConfig
+from vizion3d.annotation import ObjectMaskAnnotation3D, ObjectMaskAnnotation3DCommand
+from vizion3d.annotation.models import ObjectMaskAnnotation3DConfig
+
+stereo_result = StereoDepth().run(
+    StereoDepthCommand(
+        left_image="left.png",
+        right_image="right.png",
+        return_point_cloud=True,
+        advanced_config=StereoDepthAdvancedConfig(
+            focal_length=1733.74,
+            cx=792.27,
+            cy=541.89,
+            baseline=536.62,
+        ),
+    )
+)
+
+annotation_result = ObjectMaskAnnotation3D().run(
+    ObjectMaskAnnotation3DCommand(
+        point_cloud=stereo_result.point_cloud,
+        return_annotated_cloud=True,
+        advanced_config=ObjectMaskAnnotation3DConfig(
+            fx=1733.74,
+            fy=1733.74,
+            cx=792.27,
+            cy=541.89,
+        ),
+    )
+)
+
+for ann in annotation_result.annotations:
+    print(f"{ann.label:20s}  conf={ann.confidence:.2f}  3D points={len(ann.point_indices)}")
+
+o3d.io.write_point_cloud("annotated.ply", annotation_result.annotated_cloud)
+```
+
+See [Object Mask Annotation 3D — Stereo integration](../annotation/object_mask_annotation_3d.md#5-stereo-point-cloud-integration) for the full walkthrough.
+
+---
+
 ## Known limitations
 
-- **Rectified pairs required** — images must be stereo-rectified so corresponding points lie on the same horizontal scanline.  Un-rectified pairs will produce incorrect results.
-- **Metric scale depends on calibration** — an incorrect `baseline` or `focal_length` scales all depth values uniformly.  Always use calibrated values for real applications.
+- **Rectified pairs required** — images must be stereo-rectified so corresponding points lie on the same horizontal scanline.  Un-rectified pairs will not produce reliable results.
+- **Metric scale depends on calibration** — an inaccurate `baseline` or `focal_length` scales all depth values uniformly.  Always use calibrated values for real applications.
 - **Python 3.12 required for Open3D** — `return_depth_image` and `return_point_cloud` require Open3D, which currently only supports Python 3.12 in this project.

tests/integration/test_object_mask_annotation_3d_rest.py

@@ -70,7 +70,9 @@ def _run_group(model_backend, indoor_image_bytes, indoor_point_cloud_ply,
             ENDPOINT,
             files={
                 "image": ("scene.jpg", indoor_image_bytes, "image/jpeg"),
-                "point_cloud_ply": ("cloud.ply", indoor_point_cloud_ply, "application/octet-stream"),
+                "point_cloud_ply": (
+                    "cloud.ply", indoor_point_cloud_ply, "application/octet-stream"
+                ),
             },
             data={"model_backend": model_backend, "return_annotated_cloud": "true"},
         )

tests/unit/test_object_mask_annotation_3d_config.py

@@ -13,12 +13,12 @@
 from PIL import Image
 
 from vizion3d.annotation.commands import ObjectMaskAnnotation3DCommand
-from vizion3d.annotation.handlers import ObjectMaskAnnotation3DHandler
-from vizion3d.annotation.models import ObjectMaskAnnotation3DConfig, ObjectMaskAnnotation3DResult
 from vizion3d.annotation.defaults import (
     DEFAULT_ANNOTATION_MODEL_FILENAME,
     DEFAULT_ANNOTATION_MODEL_URL,
 )
+from vizion3d.annotation.handlers import ObjectMaskAnnotation3DHandler
+from vizion3d.annotation.models import ObjectMaskAnnotation3DConfig, ObjectMaskAnnotation3DResult
 
 o3d = pytest.importorskip("open3d", reason="open3d required")
 
@@ -116,9 +116,10 @@ def _fake(self_inner, model_path, image, cfg):
 
 class TestRESTConfigParsing:
     def test_rest_default_config_accepted(self, dummy_image_bytes, small_point_cloud):
+        from fastapi.testclient import TestClient
+
         from vizion3d.lifting.utils import create_ply_binary
         from vizion3d.server.rest.app import app
-        from fastapi.testclient import TestClient
 
         pts = np.asarray(small_point_cloud.points).astype(np.float32)
         cols = (np.asarray(small_point_cloud.colors) * 255).astype(np.uint8)
@@ -145,9 +146,9 @@ def _fake_run(*args, **kwargs):
 
 class TestGRPCConfigUnmarshalling:
     def test_grpc_default_config(self):
+        from vizion3d.lifting.utils import create_ply_binary
         from vizion3d.proto import lifting_pb2
         from vizion3d.server.grpc.server import LiftingServiceServicer
-        from vizion3d.lifting.utils import create_ply_binary
 
         pts = np.array([[0.0, 0.0, 1.0]], dtype=np.float32)
         cols = np.array([[128, 128, 128]], dtype=np.uint8)
@@ -171,9 +172,9 @@ def _fake(self_inner, cmd):
         assert received[0].conf_threshold == pytest.approx(0.25)
 
     def test_grpc_custom_config_applied(self):
+        from vizion3d.lifting.utils import create_ply_binary
         from vizion3d.proto import lifting_pb2
         from vizion3d.server.grpc.server import LiftingServiceServicer
-        from vizion3d.lifting.utils import create_ply_binary
 
         pts = np.array([[0.0, 0.0, 1.0]], dtype=np.float32)
         ply_bytes = create_ply_binary(pts, np.array([[128, 128, 128]], dtype=np.uint8))

tests/unit/test_object_mask_annotation_3d_handler.py

@@ -122,12 +122,16 @@ def test_out_of_image_filtered_out(self):
 
 
 class TestFrontViewSynthesis:
-    def test_canvas_size_derived_from_intrinsics(self, small_point_cloud):
+    def test_canvas_size_derived_from_projected_extent(self, small_point_cloud):
         pts, cols = _extract_cloud_arrays(small_point_cloud)
         cfg = ObjectMaskAnnotation3DConfig(fx=100.0, fy=100.0, cx=32.0, cy=24.0)
         img = _render_front_view(pts, cols, cfg)
-        assert img.width == int(cfg.cx * 2) + 1
-        assert img.height == int(cfg.cy * 2) + 1
+        X, Y, Z = pts[:, 0], pts[:, 1], pts[:, 2]
+        valid = Z > 0
+        u = np.round(cfg.fx * X[valid] / Z[valid] + cfg.cx).astype(np.int32)
+        v = np.round(cfg.fy * Y[valid] / Z[valid] + cfg.cy).astype(np.int32)
+        assert img.width == max(int(u.max()) + 1, 1)
+        assert img.height == max(int(v.max()) + 1, 1)
 
     def test_returns_pil_image(self, small_point_cloud):
         pts, cols = _extract_cloud_arrays(small_point_cloud)

vizion3d/lifting/handlers.py

@@ -185,9 +185,9 @@ def _run_depth_anything_checkpoint(self, model_path: str, image: Image.Image) ->
         device_type = device if isinstance(device, str) else device.type
         if device_type == "cuda":
             autocast_ctx = torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=True)
-        elif device_type == "mps":
-            autocast_ctx = torch.amp.autocast(device_type="mps", dtype=torch.float16, enabled=True)
         else:
+            # MPS float16 can produce degraded depth maps due to limited precision
+            # in transformer attention operations.  CPU has no autocast benefit.
             autocast_ctx = contextlib.nullcontext()
 
         with torch.inference_mode(), autocast_ctx: