Merge branch 'main' into feat/201-model-align

Signed-off-by: Felix Hilgers <felix.hilgers@fau.de>

Author: fhilgers

README.md

@@ -84,6 +84,14 @@ make export-yolo-onnx
 make export-midas-onnx
 ```
 
+### FP16 Quantization (Optional)
+
+Export models with FP16 precision for ~50% size reduction:
+
+```bash
+ONNX_HALF_PRECISION=true make export-onnx
+```
+
 To start the analyzer service with ONNX backend:
 ```bash
 DETECTOR_BACKEND=onnx DEPTH_BACKEND=onnx make run-analyzer-local
@@ -113,32 +121,80 @@ Available CLI flags:
 ### Environment Variables
 
 Optional environment variables:
-- `CAMERA_INDEX` (default 0) – select webcam device
-- `REGION_SIZE` (default 5) – size of the central bounding box region where we take the mean of the depth map from (should be odd for symmetry)
-- `SCALE_FACTOR` (default 432.0) – scaling of the relative depth map generated by MiDaS (must be determined empirically)
-- `CAMERA_FX/FY/CX/CY` – intrinsic matrix entries in pixels (set these when you have calibrated your camera; overrides FOV-derived values)
-- `CAMERA_FOV_X_DEG/CAMERA_FOV_Y_DEG` – fallback field of view (used only when FX/FY are not provided)
-- `DEPTH_BACKEND` – `torch` (default), `onnx`, or `depth_anything_v2`
-- `MIDAS_MODEL_TYPE` – MiDaS variant to load (`MiDaS_small`, `DPT_Hybrid`, `DPT_Large`)
-- `MIDAS_MODEL_REPO` – torch.hub repo for MiDaS (default `intel-isl/MiDaS`)
-- `DEPTH_ANYTHING_MODEL` – Hugging Face model ID for Depth Anything V2 (default `depth-anything/Depth-Anything-V2-Small-hf`)
-- `MIDAS_ONNX_MODEL_PATH` – defaults to `models/midas_small.onnx`
+- `CAMERA_INDEX` (default 0) - select webcam device
+- `REGION_SIZE` (default 5) - size of the central bounding box region where we take the mean of the depth map from (should be odd for symmetry)
+- `SCALE_FACTOR` (default 432.0) - scaling of the relative depth map generated by MiDaS (must be determined empirically)
+- `UPDATE_FREQ` (default 2) - number of frames between depth updates
+- `TARGET_SCALE_INIT` (default 0.8) - initial downscale factor for images
+- `SMOOTH_FACTOR` (default 0.15) - smoothing factor for scale updates
+- `MIN_SCALE` (default 0.2) - minimum allowed scale
+- `MAX_SCALE` (default 1.0) - maximum allowed scale
+- `FPS_THRESHOLD` (default 15.0) - threshold FPS for skipping more frames
+- `DEPTH_ANYTHING_SCALE_FACTOR` (default 0.5) - tunable Depth Anything scale factor
+- `CAMERA_FX/FY/CX/CY` - intrinsic matrix entries in pixels (set these when you have calibrated your camera; overrides FOV-derived values)
+- `CAMERA_FOV_X_DEG/CAMERA_FOV_Y_DEG` - fallback field of view (used only when FX/FY are not provided)
+- `DEPTH_BACKEND` - `torch` (default), `onnx`, or `depth_anything_v2`
+- `MIDAS_MODEL_TYPE` - MiDaS variant to load (`MiDaS_small`, `DPT_Hybrid`, `DPT_Large`)
+- `MIDAS_MODEL_REPO` - torch.hub repo for MiDaS (default `intel-isl/MiDaS`)
+- `MIDAS_CACHE_DIR` - MiDaS cache directory (default `models/midas_cache`)
+- `DEPTH_ANYTHING_MODEL` - Hugging Face model ID for Depth Anything V2 (default `depth-anything/Depth-Anything-V2-Small-hf`)
+- `DEPTH_ANYTHING_CACHE_DIR` - Depth Anything cache directory (default `models/depth_anything_cache`)
+- `MIDAS_ONNX_MODEL_PATH` - defaults to `models/midas_small.onnx`
 - `MIDAS_ONNX_INPUT_SIZE` – input size for MiDaS ONNX preprocessing (default: `384`)
-- `MIDAS_ONNX_PROVIDERS` – comma separated ONNX Runtime providers for depth (falls back to `ONNX_PROVIDERS`)
+- `MIDAS_ONNX_PROVIDERS` - comma separated ONNX Runtime providers for depth (falls back to `ONNX_PROVIDERS`)
 - `ONNX_SHARED_PREPROCESSING` – reuse one resize step for ONNX detector + depth when sizes align (default: `true`)
-- `DETECTOR_BACKEND` – `torch` (default) or `onnx`
-- `TORCH_DEVICE` – force PyTorch to use `cuda:0`, `cpu`, etc. (defaults to best available)
-- `TORCH_HALF_PRECISION` – `auto` (default), `true`, or `false`
-- `ONNX_MODEL_PATH` – defaults to `models/yolo11n.onnx`
-- `ONNX_OPSET` – opset used during ONNX export (default: 18 via `make export-onnx`)
-- `ONNX_SIMPLIFY` – simplify the exported ONNX graph (`true`/`false`, default: true)
-- `ONNX_PROVIDERS` – comma separated list such as `CUDAExecutionProvider,CPUExecutionProvider`
+- `DETECTOR_BACKEND` - `torch` (default) or `onnx`
+- `TORCH_DEVICE` - force PyTorch to use `cuda:0`, `cpu`, etc. (defaults to best available)
+- `TORCH_HALF_PRECISION` - `auto` (default), `true`, or `false`
+- `MODEL_PATH` (default `models/yolo11n.pt`) - default YOLO model path (used when no CLI flag is provided)
+- `ONNX_MODEL_PATH` - defaults to `models/yolo11n.onnx`
+- `ONNX_OPSET` - opset used during ONNX export (default: 18 via `make export-onnx`)
+- `ONNX_SIMPLIFY` - simplify the exported ONNX graph (`true`/`false`, default: true)
+- `ONNX_PROVIDERS` - comma separated list such as `CUDAExecutionProvider,CPUExecutionProvider`
 - `DETECTOR_IMAGE_SIZE`, `DETECTOR_CONF_THRESHOLD`, `DETECTOR_IOU_THRESHOLD`, `DETECTOR_MAX_DETECTIONS`, `DETECTOR_NUM_CLASSES`
-- `MODEL_PATH` (default `models/yolo11n.pt`) – default YOLO model path (used when no CLI flag is provided)
-- `VIDEO_FILE_PATH` (default `video.mp4` relative to the `/backend` folder) – default video file path for the file WebRTC service
+- `TRACKING_IOU_THRESHOLD` (default 0.1) - minimum IoU to match detection to track
+- `TRACKING_MAX_FRAMES_WITHOUT_DETECTION` (default 10) - frames before removing stale tracks
+- `TRACKING_EARLY_TERMINATION_IOU` (default 0.9) - early termination threshold for matching
+- `TRACKING_CONFIDENCE_DECAY` (default 0.1) - confidence decay per interpolation factor
+- `TRACKING_MAX_HISTORY_SIZE` (default 5) - size for history of each tracked object
+- `DETECTION_THRESHOLD` (default 2) - minimum detections before a track becomes active/sent
+- `VIDEO_FILE_PATH` (default `video.mp4` relative to the `/backend` folder) - default video file path for the file WebRTC service
+- `VIDEO_SOURCE_TYPE` (default `webcam`) - video source for the streamer (`webcam` or `file`)
+- `STREAMER_OFFER_URL` (default `http://localhost:8000/offer`) - upstream offer URL for the analyzer
+- `STUN_SERVER` (default `stun:stun.l.google.com:19302`) - STUN server for WebRTC
+- `ICE_GATHERING_TIMEOUT` (default 5.0) - timeout for ICE gathering
+- `CORS_ORIGINS` (default `*`) - comma separated CORS origins
+- `LOG_INTRINSICS` (default false) - log resolved intrinsics at runtime
+- `ANALYZER_SETTINGS_FILE` - path to JSON settings file (default `config/analyzer.json`)
 
 > Check `src/backend/common/config.py`.
 
+### Analyzer settings file (JSON)
+
+The analyzer can load a JSON settings file on startup. If the file does not
+exist, it falls back to the default config values.
+
+Default path:
+- `config/analyzer.json`
+
+Override the path:
+- `ANALYZER_SETTINGS_FILE=/path/to/analyzer.json`
+
+Format:
+- JSON object where keys match the config names in `src/backend/common/config.py`.
+- Values in the JSON override the defaults and environment variables for the analyzer.
+
+Example `config/analyzer.json`:
+```json
+{
+  "MODEL_PATH": "models/yolo11n.pt",
+  "DETECTOR_BACKEND": "onnx",
+  "DEPTH_BACKEND": "depth_anything_v2",
+  "DETECTOR_CONF_THRESHOLD": 0.35,
+  "TRACKING_IOU_THRESHOLD": 0.2
+}
+```
+
 
 ### Calibrate depth and XYZ
 - Set camera intrinsics: if you have calibrated values, export them to env vars (pixels): `CAMERA_FX`, `CAMERA_FY`, `CAMERA_CX`, `CAMERA_CY`. If not, set approximate FOVs: `CAMERA_FOV_X_DEG=78 CAMERA_FOV_Y_DEG=65` (defaults). Intrinsics are derived from the first frame size plus these values.

config/analyzer.json

@@ -0,0 +1,7 @@
+{
+  "MODEL_PATH": "models/yolo11n.pt",
+  "DETECTOR_BACKEND": "torch",
+  "DEPTH_BACKEND": "torch",
+  "DETECTOR_CONF_THRESHOLD": 0.25,
+  "TRACKING_IOU_THRESHOLD": 0.1
+}

config/analyzer.json.license

@@ -0,0 +1,3 @@
+SPDX-FileCopyrightText: 2025 robot-visual-perception
+
+SPDX-License-Identifier: MIT

scripts/download_models.py

@@ -164,7 +164,8 @@ def main() -> None:
                 output_path=yolo_onnx_target,
                 opset=args.onnx_opset,
                 imgsz=config.DETECTOR_IMAGE_SIZE,
-                simplify=args.onnx_simplify
+                simplify=args.onnx_simplify,
+                half=config.ONNX_HALF_PRECISION,
             )
 
     # --- MiDaS Processing ---
@@ -199,6 +200,7 @@ def main() -> None:
                 model_repo=args.midas_repo,
                 opset=args.onnx_opset,
                 input_size=config.MIDAS_ONNX_INPUT_SIZE,
+                half=config.ONNX_HALF_PRECISION,
             )
 
     # --- Depth Anything Processing ---

src/backend/analyzer/main.py

@@ -25,6 +25,8 @@
 from fastapi.middleware.cors import CORSMiddleware
 
 from common.config import config
+
+config.apply_settings_file(config.ANALYZER_SETTINGS_FILE)
 from common.core.detector import get_detector
 from common.core.depth import get_depth_estimator
 from analyzer.routes import router, on_shutdown

src/backend/analyzer/manager.py

@@ -274,15 +274,38 @@ async def _process_frames(self, source_track: MediaStreamTrack) -> None:
             target_scale=self.target_scale_init, source_track=source_track
         )
 
+        # Shared frame buffer coordinated via lock + event
+        latest_frame: tuple[int, np.ndarray] | None = None
+        frame_lock = asyncio.Lock()
+        frame_ready = asyncio.Event()
+
+        async def frame_receiver() -> None:
+            """Continuously receive frames and store the latest one with its id."""
+            nonlocal latest_frame
+            while self.active_connections:
+                frame_array = await self._receive_and_convert_frame(state)
+                if frame_array is None:
+                    continue
+
+                async with frame_lock:
+                    latest_frame = (state.frame_id, frame_array)
+                    frame_ready.set()
+
+        receiver_task = asyncio.create_task(frame_receiver())
+
         try:
             while self.active_connections:
                 try:
-                    frame_array = await self._receive_and_convert_frame(state)
-                    if frame_array is None:
-                        continue
+                    # Wait until a new frame is available
+                    await frame_ready.wait()
+                    frame_ready.clear()
 
-                    state.frame_id += 1
-                    state.fps_counter += 1
+                    async with frame_lock:
+                        if latest_frame is None:
+                            continue
+                        current_frame_id, frame_array = latest_frame
+                        state.frame_id = current_frame_id
+                        state.fps_counter += 1
 
                     state, current_time = self._update_fps_and_scaling(state)
                     frame_small = resize_frame(frame_array, state.target_scale)
@@ -310,6 +333,12 @@ async def _process_frames(self, source_track: MediaStreamTrack) -> None:
             logger.warning("Frame processing cancelled")
         except Exception as e:
             logger.warning("Processing task error", extra={"error": str(e)})
+        finally:
+            receiver_task.cancel()
+            try:
+                await receiver_task
+            except asyncio.CancelledError:
+                pass
 
     async def _receive_and_convert_frame(
         self, state: ProcessingState
@@ -328,6 +357,7 @@ async def _receive_and_convert_frame(
 
         try:
             frame = await asyncio.wait_for(track.recv(), timeout=5.0)
+            state.frame_id += 1
             state.consecutive_errors = 0
         except asyncio.TimeoutError:
             logger.warning("Frame receive timeout, skipping")
@@ -365,7 +395,7 @@ async def _receive_and_convert_frame(
             return None
 
         try:
-            frame_array = frame.to_ndarray(format="bgr24")  # type: ignore[union-attr]
+            frame_array = frame.to_ndarray(format="bgr24").copy()  # type: ignore[union-attr]
             return frame_array
         except AttributeError:
             logger.warning(

src/backend/common/config.py

@@ -1,8 +1,9 @@
 # SPDX-FileCopyrightText: 2025 robot-visual-perception
 #
 # SPDX-License-Identifier: MIT
+import json
 import os
-from typing import Optional
+from typing import Optional, Any
 from pathlib import Path
 
 
@@ -81,6 +82,9 @@ class Config:
     # WebRTC settings
     STUN_SERVER: str = os.getenv("STUN_SERVER", "stun:stun.l.google.com:19302")
     ICE_GATHERING_TIMEOUT: float = float(os.getenv("ICE_GATHERING_TIMEOUT", "5.0"))
+    ANALYZER_SETTINGS_FILE: Path = Path(
+        os.getenv("ANALYZER_SETTINGS_FILE", "config/analyzer.json")
+    )
 
     # Analyzer mode (for analyzer.py)
     STREAMER_OFFER_URL: str = os.getenv(
@@ -107,6 +111,11 @@ class Config:
     DETECTOR_NUM_CLASSES: int = int(os.getenv("DETECTOR_NUM_CLASSES", "80"))
     TORCH_DEVICE: Optional[str] = os.getenv("TORCH_DEVICE")
     TORCH_HALF_PRECISION: str = os.getenv("TORCH_HALF_PRECISION", "auto")
+    ONNX_HALF_PRECISION: bool = os.getenv("ONNX_HALF_PRECISION", "false").lower() in (
+        "1",
+        "true",
+        "yes",
+    )
     ONNX_PROVIDERS: list[str] = [
         provider.strip()
         for provider in os.getenv("ONNX_PROVIDERS", "").split(",")
@@ -115,6 +124,11 @@ class Config:
     ONNX_SHARED_PREPROCESSING: bool = os.getenv(
         "ONNX_SHARED_PREPROCESSING", "true"
     ).lower() in ("1", "true", "yes")
+    ONNX_IO_BINDING: bool = os.getenv("ONNX_IO_BINDING", "false").lower() in (
+        "1",
+        "true",
+        "yes",
+    )
 
     # Tracking/interpolation settings
     # Minimum IoU to match detection to track
@@ -136,5 +150,33 @@ class Config:
     # Minimum detections before a track becomes active/sent
     DETECTION_THRESHOLD: int = int(os.getenv("DETECTION_THRESHOLD", "2"))
 
+    def apply_settings_file(self, path: Path | str | None) -> bool:
+        """Apply analyzer settings from a JSON file if present."""
+        if not path:
+            return False
+        settings_path = Path(path)
+        if not settings_path.is_file():
+            return False
+        with settings_path.open("r", encoding="utf-8") as handle:
+            data = json.load(handle)
+        if not isinstance(data, dict):
+            raise ValueError("Analyzer settings file must contain a JSON object")
+        for key, value in data.items():
+            if not hasattr(self, key):
+                continue
+            current = getattr(self, key)
+            setattr(self, key, _coerce_value(value, current))
+        return True
+
+
+def _coerce_value(value: Any, current: Any) -> Any:
+    if isinstance(current, Path):
+        if value is None:
+            return value
+        return Path(value).expanduser().resolve()
+    if isinstance(current, list) and isinstance(value, str):
+        return [item.strip() for item in value.split(",") if item.strip()]
+    return value
+
 
 config = Config()

src/backend/common/core/detector.py

@@ -3,7 +3,7 @@
 # SPDX-License-Identifier: MIT
 import asyncio
 from pathlib import Path
-from typing import Optional, Callable
+from typing import Optional, Callable, Any
 import logging
 
 import numpy as np
@@ -270,12 +270,14 @@ def __init__(self, model_path: Optional[Path] = None) -> None:
         self._iou = config.DETECTOR_IOU_THRESHOLD
         self._max_det = config.DETECTOR_MAX_DETECTIONS
         self._num_classes = config.DETECTOR_NUM_CLASSES
+        self._use_io_binding = config.ONNX_IO_BINDING
+        self._io_binding: Optional[Any] = None
+        self._io_device_type, self._io_device_id = self._resolve_io_binding_device()
 
     def predict(self, frame_rgb: np.ndarray) -> list[Detection]:
         """Run ONNX Runtime inference and return scaled, filtered detections."""
         input_tensor, ratio, dwdh = self._prepare_input(frame_rgb)
-        ort_inputs = {self._input_name: input_tensor}
-        outputs = self._session.run(self._output_names, ort_inputs)[0]
+        outputs = self._run_onnx(input_tensor)[0]
         h, w = frame_rgb.shape[:2]
         return self._postprocess(outputs, (h, w), ratio, dwdh)
 
@@ -395,6 +397,58 @@ def _resolve_providers(self) -> list[str]:
         providers = [p for p in preferred if p in available]
         return providers or available
 
+    def _resolve_io_binding_device(self) -> tuple[str, int]:
+        """Pick the device type/id used for IO binding based on providers."""
+        try:
+            providers = self._session.get_providers()
+        except Exception:
+            return ("cpu", 0)
+        provider_map = {
+            "CUDAExecutionProvider": "cuda",
+            "ROCMExecutionProvider": "rocm",
+            "DmlExecutionProvider": "dml",
+        }
+        for provider in providers:
+            if provider in provider_map:
+                return (provider_map[provider], 0)
+        return ("cpu", 0)
+
+    def _run_onnx(self, input_tensor: np.ndarray) -> list[np.ndarray]:
+        """Run ONNX Runtime inference with optional IO binding."""
+        if not self._use_io_binding or self._io_device_type == "cpu":
+            ort_inputs = {self._input_name: input_tensor}
+            return self._session.run(self._output_names, ort_inputs)
+
+        if self._io_binding is None:
+            self._io_binding = self._session.io_binding()
+
+        io_binding = self._io_binding
+        if io_binding is None:
+            ort_inputs = {self._input_name: input_tensor}
+            return self._session.run(self._output_names, ort_inputs)
+
+        try:
+            io_binding.clear_binding_inputs()
+            io_binding.clear_binding_outputs()
+
+            ort_value = ort.OrtValue.ortvalue_from_numpy(
+                input_tensor, self._io_device_type, self._io_device_id
+            )
+            io_binding.bind_ortvalue_input(self._input_name, ort_value)
+            for output_name in self._output_names:
+                io_binding.bind_output(
+                    output_name, self._io_device_type, self._io_device_id
+                )
+            self._session.run_with_iobinding(io_binding)
+            return io_binding.copy_outputs_to_cpu()
+        except Exception as exc:
+            logger.warning(
+                "IO binding failed, falling back to session.run",
+                extra={"error": str(exc)},
+            )
+            ort_inputs = {self._input_name: input_tensor}
+            return self._session.run(self._output_names, ort_inputs)
+
 
 # Register built-in backends
 register_detector_backend("torch", _TorchDetector)