Seeking suggestions for evaluating the Ego4D dataset

[sdqdlgj]

Hi Kyle,

Thanks for open-sourcing the SPELL code, which demonstrates excellent performance.

May I inquire about how to evaluate the mAP performance of the Ego4D dataset?
The official Ego4D repository provides mAP evaluation code (https://github.com/EGO4D/audio-visual/tree/main/active-speaker-detection/active_speaker), but I have a few questions:

1. The official Ego4D repository presents a complex testing process, and the format of some files, such as val.txt, v.txt and etc., are unclear. Do you have any usage suggestions for the Ego4D evaluation code?
2. I would like to know whether the mAP metric for Ego4D is calculated in the same way as the mAP metric for the AVA dataset?
   Thank you in advance!

Best,
Guanjun

[kylemin]

Hi Guanjun,

1. You need to follow all the steps presented in this. All the txt files (val.txt and v.txt) will be generated automatically if you follow all the steps.
2. The evaluation metric for Ego4D is mAP@0.5, which differs from mAP for the AVA dataset. For example, if you look at Table 21 of the Ego4D paper, you can see that it reports mAP@0.5. The official evaluation script (i.e. main.py) implements the computation of mAP@0.5.

I hope this helps.
Kyle

[sdqdlgj]

Hi Kyle,

Thanks for your quick response!

I’ve reviewed the code in the main.py file, and it appears to calculate the mAP based on detected face bounding boxes (IoU > 0.5).

Given this, should both the ground truth face bounding boxes and detected face bounding boxes be required to calculate mAP@0.5? If so, I’m wondering how we can access the ground truth face bounding boxes?

Additionally, I noticed that the Ego4D ASD task only provides one set of face bounding boxes. Does this represent the ground truth face bounding boxes, or are they used for both the ground truth and detected face bounding boxes?

Best regards,
Guanjun