Merge branch 'main' into sbel/analytics-hierarchy

Author: tdorauintc

.github/resources/requirements.txt

@@ -3,7 +3,7 @@
 # This file is licensed under Apache 2.0 License.
 
 pip == 26.0.1
-pytest == 9.0.2
+pytest == 9.0.3
 numpy == 2.2.6
 selenium == 4.41.0
 selenium-wire == 5.1.0

tools/tracker/evaluation/Agents.md

@@ -21,7 +21,9 @@ SPDX-License-Identifier: Apache-2.0
 - Architecture & flow: [docs/design/tracker-evaluation-pipeline.md](../../../docs/design/tracker-evaluation-pipeline.md)
 - Main tracker evaluation README (canonical formats, usage, CLI): [README.md](README.md)
 - ADR context: [docs/adr/0009-tracking-evaluation.md](../../../docs/adr/0009-tracking-evaluation.md)
-- Example configuration: [pipeline_configs/metric_test_evaluation.yaml](pipeline_configs/metric_test_evaluation.yaml)
+- Example configurations:
+  - Full tracker evaluation: [pipeline_configs/metric_test_evaluation.yaml](pipeline_configs/metric_test_evaluation.yaml)
+  - Camera projection accuracy: [pipeline_configs/camera_projection_evaluation.yaml](pipeline_configs/camera_projection_evaluation.yaml)
 
 ## Folders structure
 
@@ -56,6 +58,19 @@ Check `datasets/README.md` for more details
   - Dependent on internal implementation: loads configuration file and calls API of SceneScape classes from scene_common and controller modules.
   - Uses separate frame ingestion logic depending on enabling time-chunking in the configuration.
 
+- **CameraProjectionHarness**: `harnesses/camera_projection_harness/camera_projection_harness.py`
+  - Bypasses the full tracker and only applies camera-pose projection to isolate per-camera calibration error.
+  - Runs `run_projection.py` inside the `scenescape-controller` Docker container (requires `scene_common`, OpenCV, open3d).
+  - Supports two projection modes per object category via the `object_classes` custom config key:
+    - **TYPE_1** (`shift_type: 1`, default): projects bounding-box bottom-centre `(centre_x, bottom_y)` to world XY plane using `CameraPose.cameraPointToWorldPoint()`.
+    - **TYPE_2** (`shift_type: 2`): shifts the projection point upward by `(height/2) * (baseAngle/90)` before projecting, using `CameraPose.projectBounds()` to derive the base angle.
+  - After projection, applies a size offset: pushes the world position `mean([x_size, y_size]) / 2` metres away from the camera, matching `MovingObject.mapObjectDetectionToWorld()`.
+  - `set_custom_config()` accepts `object_classes` (list of `{name, shift_type, x_size, y_size}` dicts) and `container_image`.
+  - `process_inputs()` serialises `object_classes` to `params.json` in the shared temp dir before launching the container; `run_projection.py` reads it at startup.
+  - `reset()` clears `_object_classes` in addition to other state.
+  - Encodes output object IDs as `"{camera_id}:{object_id}"` for downstream splitting by `CameraAccuracyEvaluator`.
+  - Pair this harness with `CameraAccuracyEvaluator` and the `camera_projection_evaluation.yaml` pipeline config.
+
 Check `harnesses/README.md` for more details
 
 ## Evaluators
@@ -69,20 +84,33 @@ Check `harnesses/README.md` for more details
 - **JitterEvaluator**: `evaluators/jitter_evaluator.py`
   Measures trajectory smoothness via RMS jerk and acceleration variance, computed from both tracker outputs and ground-truth tracks. Supports GT and ratio variants to isolate tracker-added jitter from dataset-inherent jitter.
 
+- **CameraAccuracyEvaluator**: `evaluators/camera_accuracy_evaluator.py`
+  Consumes output from `CameraProjectionHarness` and measures per-camera, per-object projection accuracy:
+  - `DIST_T`: mean Euclidean distance error (m) between projected and GT world position per (camera, object) pair.
+  - `VISIBILITY`: frame count and percentage each camera detects each object.
+  - `set_scene_config()` resolves each camera's world position (`_solve_camera_position`: `cv2.solvePnP` → `C = -R^T @ t`) and 2-D viewing direction (`_solve_camera_view_dir`: `R^T @ [0, 0, 1]`, XY normalized) from the scene's calibration data.
+  - `trajectories_{cam}.png` includes a star marker at the camera position and an arrow showing its view direction; both X and Y axes are flipped when `cam_y > mean(gt_y)` (180° rotation so camera always appears at visual bottom with correct chirality).
+  - Writes `distance_errors.csv`, `visibility_summary.csv`, `accuracy_summary.csv`, `summary_table.csv` (human-readable column names), per-camera `distance_errors_{cam}.png`, `trajectories_{cam}.png`, and `error_vs_cam_distance_{cam}.png` plots, and a `visibility_bar_chart.png`. `format_summary()` returns a terminal-ready table.
+
 Multiple evaluators can be configured in a single YAML pipeline; each runs independently against the same tracker outputs and writes results to its own subfolder under the run output directory.
 
 Check `evaluators/README.md` for more details
 
 ## Code Entry Points
 
 - **Pipeline orchestration**: [pipeline_engine.py](pipeline_engine.py) (methods `load_configuration()`, `run()`, `evaluate()`, CLI via `python -m pipeline_engine <config>`).
+  - `_configure_harness()` forwards `object_classes` from the YAML `harness.config` block to the harness via `set_custom_config({'object_classes': ...})`.
+  - `_configure_evaluators()` calls `set_scene_config(scene_config)` on each evaluator that exposes the method (checked via `hasattr`), passing the scene config returned by `dataset.get_scene_config()`.
+  - `main()` prints evaluator results using `evaluator.format_summary()` when available; otherwise falls back to printing each metric value individually.
 - **Component base classes** (implement to extend pipeline):
   - Dataset: [base/tracking_dataset.py](base/tracking_dataset.py)
   - Harness: [base/tracker_harness.py](base/tracker_harness.py)
   - Evaluator: [base/tracker_evaluator.py](base/tracker_evaluator.py)
 - **TrackEval adapter & helpers**: [evaluators/trackeval_evaluator.py](evaluators/trackeval_evaluator.py), [utils/format_converters/](./utils/format_converters.py).
 - **Jitter adapter**: [evaluators/jitter_evaluator.py](evaluators/jitter_evaluator.py).
 - **Diagnostic adapter**: [evaluators/diagnostic_evaluator.py](evaluators/diagnostic_evaluator.py).
+- **Camera projection harness**: [harnesses/camera_projection_harness/camera_projection_harness.py](harnesses/camera_projection_harness/camera_projection_harness.py), container script: [harnesses/camera_projection_harness/run_projection.py](harnesses/camera_projection_harness/run_projection.py).
+- **Camera accuracy evaluator**: [evaluators/camera_accuracy_evaluator.py](evaluators/camera_accuracy_evaluator.py).
 
 ## Guidelines for Adding New Component or Updating Existing One
 

tools/tracker/evaluation/README.md

@@ -50,6 +50,8 @@ pip install -r requirements.txt
 
 Create a YAML configuration file (see `pipeline_configs/` directory):
 
+**Full tracker evaluation** (`pipeline_configs/controller_evaluation.yaml`):
+
 ```yaml
 pipeline:
   output:
@@ -59,7 +61,7 @@ dataset:
   class: datasets.metric_test_dataset.MetricTestDataset
   config:
     data_path: /path/to/dataset
-    cameras: [x1, x2]
+    cameras: [Cam_x1_0, Cam_x2_0]
     camera_fps: 30
 
 harness:
@@ -88,6 +90,42 @@ evaluators:
         ]
 ```
 
+**Camera projection accuracy** (`pipeline_configs/camera_projection_evaluation.yaml`):
+
+Bypasses the tracker and only applies camera-pose projection to isolate per-camera calibration error:
+
+```yaml
+pipeline:
+  output:
+    path: /tmp/camera-projection-evaluation
+
+dataset:
+  class: datasets.metric_test_dataset.MetricTestDataset
+  config:
+    data_path: /path/to/dataset
+    cameras: [Cam_x1_0, Cam_x2_0]
+    camera_fps: 30
+
+harness:
+  class: harnesses.camera_projection_harness.CameraProjectionHarness
+  config:
+    container_image: scenescape-controller:latest
+    # Optional: per-category projection settings.
+    # shift_type 1 = bottom-centre (TYPE_1, default)
+    # shift_type 2 = perspective-corrected point (TYPE_2)
+    # x_size / y_size push the result away from the camera by mean([x,y])/2 metres.
+    object_classes:
+      - name: person
+        shift_type: 1
+        x_size: 0.5
+        y_size: 0.5
+
+evaluators:
+  - class: evaluators.camera_accuracy_evaluator.CameraAccuracyEvaluator
+    config:
+      metrics: [DIST_T, VISIBILITY]
+```
+
 Run the pipeline:
 
 ```bash
@@ -144,11 +182,12 @@ evaluation/
 
 ### Available Evaluators
 
-| Evaluator             | Metrics                                                                                                                         | Description                                                                                                                                                           |
-| --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `TrackEvalEvaluator`  | HOTA, MOTA, IDF1, and more                                                                                                      | Industry-standard tracking accuracy metrics via the TrackEval library                                                                                                 |
-| `DiagnosticEvaluator` | `LOC_T_X`, `LOC_T_Y`, `DIST_T` → summary scalars: `DIST_T_mean`, `LOC_T_X_mae`, `LOC_T_Y_mae`, `num_matches`                    | Per-frame location and distance error between matched tracker output tracks and ground-truth tracks; uses bipartite (Hungarian) assignment over overlapping frames    |
-| `JitterEvaluator`     | `rms_jerk`, `rms_jerk_gt`, `rms_jerk_ratio`, `acceleration_variance`, `acceleration_variance_gt`, `acceleration_variance_ratio` | Trajectory smoothness metrics based on numerical differentiation of 3D positions; GT and ratio variants allow comparing tracker-added jitter against test-data jitter |
+| Evaluator                 | Metrics                                                                                                                         | Description                                                                                                                                                           |
+| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `TrackEvalEvaluator`      | HOTA, MOTA, IDF1, and more                                                                                                      | Industry-standard tracking accuracy metrics via the TrackEval library                                                                                                 |
+| `DiagnosticEvaluator`     | `LOC_T_X`, `LOC_T_Y`, `DIST_T` → summary scalars: `DIST_T_mean`, `LOC_T_X_mae`, `LOC_T_Y_mae`, `num_matches`                    | Per-frame location and distance error between matched tracker output tracks and ground-truth tracks; uses bipartite (Hungarian) assignment over overlapping frames    |
+| `JitterEvaluator`         | `rms_jerk`, `rms_jerk_gt`, `rms_jerk_ratio`, `acceleration_variance`, `acceleration_variance_gt`, `acceleration_variance_ratio` | Trajectory smoothness metrics based on numerical differentiation of 3D positions; GT and ratio variants allow comparing tracker-added jitter against test-data jitter |
+| `CameraAccuracyEvaluator` | `DIST_T` → `dist_mean_all`, `dist_mean_{cam}`, `dist_mean_{cam}_{obj}`; `VISIBILITY` → `visibility_{cam}_{obj}` (frames + %)    | Per-camera, per-object projection accuracy: mean distance error and visibility frame count. Designed to pair with `CameraProjectionHarness`.                          |
 
 ## Canonical Data Formats
 
@@ -221,7 +260,8 @@ The evaluation pipeline has comprehensive test coverage:
 
 - **Unit Tests**: Fast tests without external dependencies, located in component-specific test directories
   - `datasets/tests/test_*.py`: Datasets unit tests
-  - `harnesses/tests/test_*.py`: Harnesses unit tests
+  - `harnesses/tests/test_*.py`: Harnesses unit tests (includes `CameraProjectionHarness` — 23 tests; `run_projection.py` helpers — 7 tests)
+  - `evaluators/tests/test_*.py`: Evaluator unit tests (includes `CameraAccuracyEvaluator` — 37 tests)
   - `tests/test_format_converters.py`: Format converter unit tests
 
 - **Integration Tests**: Tests requiring Docker and real components, located in `tests/`

tools/tracker/evaluation/datasets/README.md

@@ -25,7 +25,7 @@ Dataset adapters convert dataset-specific formats to SceneScape canonical format
 
 **Key Features**:
 - Single scene: `Retail_Demo`
-- Two cameras: `x1`, `x2` (Cam_x1_0, Cam_x2_0)
+- Two cameras: `Cam_x1_0`, `Cam_x2_0`
 - Multiple FPS options: 1, 10, 30 (separate JSON files per FPS)
 - Ground truth in MOTChallenge 3D CSV format (see [Canonical Data Formats](../README.md#canonical-data-formats))
 
@@ -42,13 +42,13 @@ from datasets.metric_test_dataset import MetricTestDataset
 dataset = MetricTestDataset("../../../tests/system/metric/dataset")
 
 # Configure dataset
-dataset.set_cameras(["x1", "x2"]).set_camera_fps(30)
+dataset.set_cameras(["Cam_x1_0", "Cam_x2_0"]).set_camera_fps(30)
 
 # Get scene configuration
 scene_config = dataset.get_scene_config()
 
 # Get camera inputs
-for camera_input in dataset.get_inputs("x1"):
+for camera_input in dataset.get_inputs("Cam_x1_0"):
     # Process detection data
     pass
 

tools/tracker/evaluation/datasets/metric_test_dataset.py

@@ -21,15 +21,15 @@ class MetricTestDataset(TrackingDataset):
 
   This dataset contains:
   - Scene: Retail_Demo (single built-in scene)
-  - Cameras: x1, x2 (Cam_x1_0, Cam_x2_0)
+  - Cameras: Cam_x1_0, Cam_x2_0
   - FPS options: 1, 10, 30 (separate JSON files per FPS)
   - Ground truth: gtLoc.json with object locations
   - Scene config: config.json with camera calibration
   """
 
   # Constants
   SCENE_NAME = "Retail_Demo"
-  SUPPORTED_CAMERAS = ["x1", "x2"]
+  SUPPORTED_CAMERAS = ["Cam_x1_0", "Cam_x2_0"]
   SUPPORTED_FPS = [1, 10, 30]
   DEFAULT_FPS = 30
 
@@ -74,7 +74,7 @@ def set_cameras(self, cameras: Optional[List[str]] = None) -> 'MetricTestDataset
     """Set cameras to use.
 
     Args:
-      cameras: List of camera IDs (subset of ["x1", "x2"])
+      cameras: List of camera IDs (subset of ["Cam_x1_0", "Cam_x2_0"])
 
     Returns:
       Self for method chaining
@@ -382,7 +382,7 @@ def _get_input_filename(self, cam_id: str) -> Path:
       raise ValueError(f"Camera {cam_id} not in configured cameras")
 
     fps_suffix = f"_{int(self._camera_fps)}fps" if self._camera_fps != 30 else ""
-    input_file = self._dataset_path / f"Cam_{cam_id}_0{fps_suffix}.json"
+    input_file = self._dataset_path / f"{cam_id}{fps_suffix}.json"
 
     if not input_file.exists():
       raise FileNotFoundError(f"Input file not found: {input_file}")