[ITEP-89467] Tracking Evaluation Pipeline: Add jitter metrics (RMS jerk and acceleration variance)

tools/tracker/evaluation/README.md

@@ -72,6 +72,17 @@ evaluators:
   - class: evaluators.trackeval_evaluator.TrackEvalEvaluator
     config:
       metrics: [HOTA, MOTA, IDF1]
+  - class: evaluators.jitter_evaluator.JitterEvaluator
+    config:
+      metrics:
+        [
+          rms_jerk,
+          rms_jerk_gt,
+          rms_jerk_ratio,
+          acceleration_variance,
+          acceleration_variance_gt,
+          acceleration_variance_ratio,
+        ]
 ```
 
 Run the pipeline:
@@ -88,10 +99,38 @@ python -m pipeline_engine config.yaml
       ├── dataset/                     # Dataset-specific caches or exports
       ├── harness/                     # Harness logs or artifacts
       └── evaluators/
-          └── <evaluator-class-name>/  # Evaluated metrics
+          └── <evaluator-key>/         # One folder per evaluator
+```
+
+The `<evaluator-key>` is the evaluator class name (e.g., `TrackEvalEvaluator`). When two evaluators
+share the same class name, an index suffix is appended to keep keys unique
+(e.g., `TrackEvalEvaluator_0/`, `TrackEvalEvaluator_1/`).
+
+Example with a single evaluator:
+
+```
+/tmp/tracker-evaluation/20260211_142530/evaluators/TrackEvalEvaluator/
+```
+
+Example with two evaluators of the same class:
+
+```
+/tmp/tracker-evaluation/20260211_142530/evaluators/TrackEvalEvaluator_0/
+/tmp/tracker-evaluation/20260211_142530/evaluators/TrackEvalEvaluator_1/
 ```
 
-Example: `/tmp/tracker-evaluation/20260211_142530/evaluators/TrackEvalEvaluator/`
+**Multiple evaluators**: The `evaluators` list accepts any number of entries. Each evaluator runs
+against the same tracker outputs independently:
+
+```yaml
+evaluators:
+  - class: evaluators.trackeval_evaluator.TrackEvalEvaluator
+    config:
+      metrics: [HOTA, MOTA]
+  - class: evaluators.trackeval_evaluator.TrackEvalEvaluator
+    config:
+      metrics: [IDF1]
+```
 
 ## Directory Structure
 
@@ -123,6 +162,13 @@ evaluation/
 1. Create a new file in `evaluators/` (e.g., `custom_evaluator.py`)
 2. Implement the `TrackerEvaluator` ABC from `base/tracker_evaluator.py`
 
+### Available Evaluators
+
+| Evaluator            | Metrics                                                                                                                         | Description                                                                                                                                                           |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `TrackEvalEvaluator` | HOTA, MOTA, IDF1, and more                                                                                                      | Industry-standard tracking accuracy metrics via the TrackEval library                                                                                                 |
+| `JitterEvaluator`    | `rms_jerk`, `rms_jerk_gt`, `rms_jerk_ratio`, `acceleration_variance`, `acceleration_variance_gt`, `acceleration_variance_ratio` | Trajectory smoothness metrics based on numerical differentiation of 3D positions; GT and ratio variants allow comparing tracker-added jitter against test-data jitter |
+
 ## Canonical Data Formats
 
 The pipeline uses standardized data formats defined by JSON schemas to enable interoperability between components. All implementations must conform to these canonical formats.
@@ -244,7 +290,7 @@ pytest . -v -m integration
 pytest tests/ -v                     # Integration tests
 pytest datasets/tests/ -v            # Dataset unit tests
 pytest harnesses/tests/ -v           # Harness unit tests
-pytest evaluators/tests/ -v           # Evaluators unit tests
+pytest evaluators/tests/ -v          # Evaluators unit tests
 ```
 
 **Run tests from a specific file**:

tools/tracker/evaluation/base/tracker_harness.py

@@ -4,7 +4,7 @@
 """Base class for tracker harness implementations."""
 
 from abc import ABC, abstractmethod
-from typing import Iterator, Dict, Any
+from typing import Dict, Any, Iterator, List
 from pathlib import Path
 
 
@@ -71,7 +71,7 @@ def set_output_folder(self, path: Path) -> 'TrackerHarness':
     pass
 
   @abstractmethod
-  def process_inputs(self, inputs: Iterator[Dict[str, Any]]) -> Iterator[Dict[str, Any]]:
+  def process_inputs(self, inputs: Iterator[Dict[str, Any]]) -> List[Dict[str, Any]]:
     """Process input detections through the tracker synchronously.
 
     This is the default (synchronous) mode. Processes all inputs and returns outputs.
@@ -82,7 +82,7 @@ def process_inputs(self, inputs: Iterator[Dict[str, Any]]) -> Iterator[Dict[str,
         (see tools/tracker/evaluation/README.md#canonical-data-formats).
 
     Returns:
-      Iterator of tracker outputs in canonical Tracker Output Format.
+      List of tracker outputs in canonical Tracker Output Format.
 
     Raises:
       RuntimeError: If processing fails.

tools/tracker/evaluation/evaluators/README.md

@@ -96,6 +96,48 @@ print(f"IDF1: {metrics['IDF1']:.3f}")
 
 **Tests**: See [tests/test_trackeval_evaluator.py](tests/test_trackeval_evaluator.py) for comprehensive test suite with 16 test cases covering configuration, processing, evaluation, and integration workflows.
 
+### JitterEvaluator
+
+**Purpose**: Evaluate tracker smoothness by measuring positional jitter in tracked object trajectories, and compare it against jitter already present in the ground-truth test data.
+
+**Status**: **FULLY IMPLEMENTED** — Computes RMS jerk and acceleration variance from both tracker outputs and ground-truth tracks using numerical differentiation.
+
+**Supported Metrics**:
+
+| Metric                        | Source         | Description                                                                  |
+| ----------------------------- | -------------- | ---------------------------------------------------------------------------- |
+| `rms_jerk`                    | Tracker output | RMS jerk across all tracker output tracks (m/s³)                             |
+| `acceleration_variance`       | Tracker output | Variance of acceleration magnitudes across all tracker output tracks (m/s²)² |
+| `rms_jerk_gt`                 | Ground truth   | Same as `rms_jerk` computed on ground-truth tracks                           |
+| `acceleration_variance_gt`    | Ground truth   | Same as `acceleration_variance` computed on ground-truth tracks              |
+| `rms_jerk_ratio`              | Tracker / GT   | `rms_jerk` / `rms_jerk_gt` — tracker jitter relative to GT (1.0 = equal)     |
+| `acceleration_variance_ratio` | Tracker / GT   | `acceleration_variance` / `acceleration_variance_gt`                         |
+
+Comparing `rms_jerk` with `rms_jerk_gt` shows how much jitter the tracker
+adds on top of any jitter already present in the test data.
+
+**Algorithm**:
+
+All metrics are derived by applying three sequential layers of forward finite differences to 3D positions, accounting for variable time steps between frames:
+
+$$v_i = \frac{p_{i+1} - p_i}{\Delta t_i}, \quad a_i = \frac{v_{i+1} - v_i}{\Delta t_{v,i}}, \quad j_i = \frac{a_{i+1} - a_i}{\Delta t_{a,i}}$$
+
+- **rms_jerk / rms_jerk_gt**: $\sqrt{\frac{1}{N}\sum |j_i|^2}$ over all jerk samples from all tracks.
+- **acceleration_variance / acceleration_variance_gt**: $\text{Var}(|a_i|)$ over all acceleration magnitude samples from all tracks.
+- **rms_jerk_ratio / acceleration_variance_ratio**: tracker metric divided by the corresponding GT metric. Returns 0.0 when the GT denominator is zero. Values >1.0 indicate the tracker adds more jitter than is inherent in the ground truth.
+
+Minimum track length: 3 points for acceleration, 4 points for jerk. Shorter tracks are skipped; if no eligible tracks exist, the metric returns 0.0.
+
+For GT metrics, ground-truth frame numbers are converted to relative timestamps using the FPS derived from the tracker output.
+
+**Key Features**:
+
+- Builds per-track position histories from canonical tracker output format.
+- Parses MOTChallenge 3D CSV ground-truth file for GT metric computation.
+- Supports variable frame rates — time deltas are computed from actual timestamps.
+- Deduplicates frames with identical timestamps (mirrors `TrackEvalEvaluator` behaviour).
+- Sorts each track's positions by timestamp before metric computation.
+- Saves a plain-text `jitter_results.txt` summary to the configured output folder.
 ### DiagnosticEvaluator
 
 **Purpose**: Per-frame location comparison and error analysis between matched output tracks and ground-truth tracks.
@@ -121,6 +163,44 @@ print(f"IDF1: {metrics['IDF1']:.3f}")
 **Usage Example**:
 
 ```python
+from pathlib import Path
+from evaluators.jitter_evaluator import JitterEvaluator
+
+evaluator = JitterEvaluator()
+evaluator.configure_metrics(['rms_jerk', 'rms_jerk_gt', 'rms_jerk_ratio',
+                             'acceleration_variance', 'acceleration_variance_gt',
+                             'acceleration_variance_ratio'])
+evaluator.set_output_folder(Path('/path/to/results'))
+
+# Pass ground_truth=None to skip GT metrics
+evaluator.process_tracker_outputs(tracker_outputs, ground_truth=dataset.get_ground_truth())
+metrics = evaluator.evaluate_metrics()
+
+print(f"RMS Jerk (tracker): {metrics['rms_jerk']:.4f} m/s³")
+print(f"RMS Jerk (GT):      {metrics['rms_jerk_gt']:.4f} m/s³")
+print(f"RMS Jerk ratio:     {metrics['rms_jerk_ratio']:.4f}  (1.0 = equal jitter)")
+```
+
+**Pipeline Configuration**:
+
+```yaml
+evaluators:
+  - class: evaluators.jitter_evaluator.JitterEvaluator
+    config:
+      metrics:
+        [
+          rms_jerk,
+          rms_jerk_gt,
+          rms_jerk_ratio,
+          acceleration_variance,
+          acceleration_variance_gt,
+          acceleration_variance_ratio,
+        ]
+```
+
+**Implementation**: [jitter_evaluator.py](jitter_evaluator.py)
+
+**Tests**: See [tests/test_jitter_evaluator.py](tests/test_jitter_evaluator.py).
 from evaluators.diagnostic_evaluator import DiagnosticEvaluator
 from pathlib import Path
 

tools/tracker/evaluation/evaluators/__init__.py

@@ -3,6 +3,7 @@
 
 """Evaluator implementations for tracker evaluation."""
 from .trackeval_evaluator import TrackEvalEvaluator
+from .jitter_evaluator import JitterEvaluator
 from .diagnostic_evaluator import DiagnosticEvaluator
 
-__all__ = ['TrackEvalEvaluator', 'DiagnosticEvaluator']
+__all__ = ['TrackEvalEvaluator', 'DiagnosticEvaluator', 'JitterEvaluator']