Extended Keypoint Metrics

[dtronmans]

Purpose

• Accepts sigmas and area_factor params, these should be the same as the ones trained with
• Accepts arbitrary number of classes and arbitrary number of keypoints per class

Example:

metrics:
  metrics:
    - name: ExtendedKeypointMetrics
      params:
        kpt_oks_sigmas: [0.08, 0.08, 0.08, 0.08]
        area_factor: 1.0

Validated with two different use cases (COCO 17 keypoints and non-COCO) and the ONNX scores match the luxonis_train test .ckpt scores

Specification

Dependencies & Potential Impact

Deployment Plan

Testing & Validation

[klemen1999]

Overall I think we'd want to do a bit more work on this before mainlined.
There are 3 different metric types to evaluate instance pose models:

• mAP where GT and predictions are matched by bounding boxes (IoU)
• mAP where GT and predictions are matchd by keypoints (OKS)
• OKS metric as a standalone

We already have mAP based on bounding boxes (here).
The current KeypointMeanAveragePrecision does the mAP based on OKS but it has limitations with number of classes which the new Extended Keypoint Metrics is solving. But these two I think can be merged together into one metric, the Extended I believe can be made in a way that it does everything that the current one does + extra features (sigmas, different number of keypoints, etc) - similar as we do in luxonis-train. One feature that the current one does have which would need to be ported to "Extended" one is that area and scale can be optionally computed from keypoints directly (here) - this makes the metric work also if model doesn't predict bbox (e.g. Lite-HRNet)
And lastly we would like to have the OKS metric as standalone. But this one can be handled in a different PR.

[dtronmans]

Overall I think we'd want to do a bit more work on this before mainlined. There are 3 different metric types to evaluate instance pose models:

• mAP where GT and predictions are matched by bounding boxes (IoU)
• mAP where GT and predictions are matchd by keypoints (OKS)
• OKS metric as a standalone

We already have mAP based on bounding boxes (here). The current KeypointMeanAveragePrecision does the mAP based on OKS but it has limitations with number of classes which the new Extended Keypoint Metrics is solving. But these two I think can be merged together into one metric, the Extended I believe can be made in a way that it does everything that the current one does + extra features (sigmas, different number of keypoints, etc) - similar as we do in luxonis-train. One feature that the current one does have which would need to be ported to "Extended" one is that area and scale can be optionally computed from keypoints directly (here) - this makes the metric work also if model doesn't predict bbox (e.g. Lite-HRNet) And lastly we would like to have the OKS metric as standalone. But this one can be handled in a different PR.

It seems to me as though what was missing from ExtendedKeypointMetrics that was not present was just the option to compute the area from keypoints instead of the provided bounding box. I've added this now, and it can be called like this:

metrics:

• name: ExtendedKeypointMetrics
  params:
  iou_type: keypoints
  compute_area_from_keypoints: true

The difference from the reference implementation KeypointMeanAveragePrecision though is that now compute_area_from_keypoints with there is scaling by area_factor, I think this makes sense for consistency sake (using bboxes instead of inferring the area from the keypoints is already being scaled by area_factor)