Ham2Pose-Style Evaluation with precision, mAP

[cleong110]

We collect all samples for every unique sign s, and randomly sample 4 × |s| other signs from the dataset to keep a 1 : 4 ratio of same : other samples. Then, we measure the distance from the reference sign to each one of the samples using nDTW-MJE as explained in Sec. 7.1, and
calculate mAP (mean Average Precision) and mean precision@k for k = 1, 5, 10, across all signs, measuring how many of the k most similar poses were of a video signing the query sign

https://arxiv.org/pdf/2211.13613

So we can implement something like this:

df # has Pose paths, Gloss, embedding paths, video paths
# make a distance cache, indexed by metric name and video IDs?
for each gloss in df:
    # get all rows for that gloss
    # randomly sample 4x as many with other glosses
    items_for_metrics_eval # has 1/5 in-gloss, 4/5 out
    for each metric:
        # calculate all the pairwise scores
        # what are the in-gloss scores  
        # what are the other-gloss scores   

Then we can calculate precision@k (given 1/5 in-class)

[AmitMY]

here is the exact Ham2Pose implementaiton: https://github.com/rotem-shalev/Ham2Pose/blob/main/test.py#L102-L148

[cleong110]

We ended up implementing this for our recent paper, on this branch.

• Scripts to build a CSV with all the relevant .pose file paths: https://github.com/cleong110/pose-evaluation/tree/datasets_to_dataframes/pose_evaluation/evaluation/dataset_parsing
• script to load the CSVs for all the datasets, and run eval: https://github.com/cleong110/pose-evaluation/blob/datasets_to_dataframes/pose_evaluation/evaluation/load_splits_and_run_metrics.py