NMS, [email protected], CM calculations

[burakocamis]

Hello, i'd like to ask some questions about the metric computation part of the algorithm. Also, i need confirmation for clear understanding. Thanks in advance for reading such a long entry and replying.

1. In order to evaluate my trained model performance for some test images, i am using val.py so that i can get [email protected] metric for each classes. The way i am understanding the computation of [email protected] is that, first, test images are inputted to the algorithm and it outputs bboxes with their class labels and confidence scores. NMS eliminates some bboxes overlapping by using NMS IoU threshold in val.py. After elimination, remaining bboxes are used for metric computation. I think NMS elimination is done for different bboxes showing the same object. Is that all true?
2. Second, for the [email protected] metric, remaining bboxes having at least %50 IoU with ground truth is selected for P-R curve calculation. Is that true?
3. Third, P-R curve is calculated over 1000 different confidence threshold values. For each confidence threshold value, the confidence scores of remaining bboxes, having at least %50 IoU, are compared with that confidence threshold value. If confidence score is greater than that confidence threshold then P and R are calculated for those detections. These steps are repeated 1000 different confidence threshold values then P-R curve is finally obtained. The area under the P-R curve for each classes gives the [email protected] values. Is that all true?
4. Same val.py calculates confusion matrix along with [email protected] metric. Confusion matrix in val.py is using 0.25 confidence threshold value. It shows that remaining bboxes having at least %50(?) IoU with ground truth satisfying 0.25 confidence scores are considered for CM calculation. Is that true?

[glenn-jocher]

@burakocamis yes all this is correct. To get the best mAP you theoretically want to compute all the way down to --conf 0.0, but 0.001 is a good approximation.

Deployed threshold default is 0.25, this is used for confusion matrix.

[burakocamis]

@glenn-jocher thank you very much