Skip to content

Commit 2005584

Browse files
authored
[ITEP-83115] Tracker Evaluation Pipeline MVP (#987)
Lay the foundation for the long-term evolution of tracker evaluation as described in Architecture Decision Record 009. Integrate the following three components into evaluation pipeline that allows to realize Phase 1 of the strategy: dataset: a simple dataset used in metric tests tracker harness: production tracker implementation based on scene-controller image metrics evaluator: TrackEval library wrapper Out of scope of this PR (will be added in separate pull requests): Mapping of metric tests to new metrics produced by TrackEval along with investigation of significant discrepancies Integrating tracker evaluation pipeline into CI Removal of legacy metric tests as redundant Adding jitter metrics evaluator into the pipeline
1 parent 5f0496c commit 2005584

53 files changed

Lines changed: 5481 additions & 29 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/resources/.prettierignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,5 @@
1313
**/CMakePresets.json
1414
**/CMakeFiles/
1515

16-
**/tests/api/README.md
16+
**/tests/api/README.md
17+
**/tests/system/metric/dataset/*.json

tests/system/metric/conftest.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,21 +37,21 @@ def params(request):
3737
@returns params Dict of parameters
3838
"""
3939
dir = os.path.dirname(os.path.abspath(__file__))
40-
input_cam_1 = os.path.join(dir, "test_data/Cam_x1_0.json")
41-
input_cam_2 = os.path.join(dir, "test_data/Cam_x2_0.json")
40+
input_cam_1 = os.path.join(dir, "dataset/Cam_x1_0.json")
41+
input_cam_2 = os.path.join(dir, "dataset/Cam_x2_0.json")
4242
params = {}
4343
params["metric"] = request.config.getoption("--metric")
4444
params["threshold"] = request.config.getoption("--threshold")
4545
params["camera_frame_rate"] = request.config.getoption("--camera_frame_rate")
4646
params["default_camera_frame_rate"] = 30
4747
params["input"] = [input_cam_1, input_cam_2]
48-
params["config"] = os.path.join(dir, "test_data/config.json")
49-
params["ground_truth"] = os.path.join(dir, "test_data/gtLoc.json")
48+
params["config"] = os.path.join(dir, "dataset/config.json")
49+
params["ground_truth"] = os.path.join(dir, "dataset/gtLoc.json")
5050
params["rootca"] = "/run/secrets/certs/scenescape-ca.pem"
5151
params["auth"] = "/run/secrets/controller.auth"
5252
params["mqtt_broker"] = "broker.scenescape.intel.com"
5353
params["mqtt_port"] = 1883
54-
params["trackerconfig"] = os.path.join(dir, "test_data", request.param)
54+
params["trackerconfig"] = os.path.join(dir, "dataset", request.param)
5555

5656
if "time-chunking" in request.param:
5757
params["trackerconfig_name"] = "time-chunking"
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.

0 commit comments

Comments
 (0)