docstring for `heatmaps_to_keypoints` does not match return values.

[WillieMaddox]

The docstring for the function heatmaps_to_keypoints in torchvision/models/detection/roi_heads.py does not match the return values.

def heatmaps_to_keypoints(maps, rois):
    """Extract predicted keypoint locations from heatmaps. Output has shape
    (#rois, 4, #keypoints) with the 4 rows corresponding to (x, y, logit, prob)
    for each keypoint.
    """
...
return xy_preds.permute(0, 2, 1), end_scores

The function returns 2 tensors. The first tensor has the shape (#rois, #keypoints, 3) with the last dimension corresponding to (x, y, vis). The second tensor has shape (#rois, #keypoints) the values of which correspond to... logits? Some explanation on what these end_scores are used for and/or how to use them would also be helpful.

[NicolasHug]

Thanks for the report @WillieMaddox , indeed the doc looks wrong. It's quite possible that it got out-dated with time.

The second tensor has shape (#rois, #keypoints) the values of which correspond to... logits? Some explanation on what these end_scores are used for and/or how to use them would also be helpful

I can't tell immediately whether these are logits or some transformed version of that, but I believe they are directly passed-through the COCO evaluation API as confidence scores: https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#LL41C27-L41C37. Another intermediate pointer to help: https://github.com/pytorch/vision/blob/6a85ef24ba7a5d5879c4bd795972881ef5001c9b/references/detection/engine.py#L102:L102

[NicolasHug]

indeed the doc looks wrong.

Would you like to submit a PR to fix them? Otherwise no worries, we'll take care of it

[WillieMaddox]

I believe the 'scores' passed to the COCO evaluation API are the bbox ['scores'] not the ['keypoints_scores']. I can't find anywhere in the torchvision or the pycocotools codebase where ['keypoints_scores'] are used.

user@machine:~/
$ lla git/coco/PythonAPI/
drwxrwxr-x user user 4.0 KB Mon Jan 30 15:37:13 2023 ./
drwxrwxr-x user user 4.0 KB Mon Jan 30 15:37:13 2023 ../
.rw-rw-r-- user user 199 B  Thu Mar  9 21:47:21 2017 Makefile
.rw-rw-r-- user user 1.7 MB Sun Nov 12 07:40:20 2017 pycocoDemo.ipynb
.rw-rw-r-- user user 4.4 KB Thu Mar  9 21:47:21 2017 pycocoEvalDemo.ipynb
drwxrwxr-x user user 4.0 KB Mon Jan 30 15:37:13 2023 pycocotools/
.rw-rw-r-- user user 762 B  Mon Jan 30 15:37:13 2023 setup.py
user@machine:~/
$ grep -nir keypoints_scores git/coco/PythonAPI/
user@machine:~/
$ grep -nir keypoints_scores git/pytorch/vision/
git/pytorch/vision/torchvision/models/detection/roi_heads.py:873:                    r["keypoints_scores"] = kps
Binary file git/pytorch/vision/test/expect/ModelTester.test_keypointrcnn_resnet50_fpn_expect.pkl matches

Unless I'm reading it wrong, in /pycocotools/cocoeval.py:

def computeOks(self, imgId, catId):
        p = self.params
        # dimention here should be Nxm
        gts = self._gts[imgId, catId]
        dts = self._dts[imgId, catId]
        inds = np.argsort([-d['score'] for d in dts], kind='mergesort')

The d['score'] is the bbox score.

[WillieMaddox]

Looks like heatmaps_to_keypoints originally came from here,

https://github.com/facebookresearch/Detectron/blob/1809dd41c1ffc881c0d6b1c16ea38d08894f8b6d/detectron/utils/keypoints.py#L103