Skip to content

Add evaluation scripts (Updated on 2023-11-07) #119

Open
@fengyuentau

Description

@fengyuentau

We now have over 15 models covering more than 10 tasks in the zoo. Although most of the models are converted to ONNX straightly from its original format, such conversion may potentially lead to drop of accuracy, especially for FP16 and Int8-quantized models. To show the actual accuracy for our users, we now already have some evaluation scripts with the following conditions met in https://github.com/opencv/opencv_zoo/tree/master/tools/eval:

  1. Reproduce the claimed accuracy with the converted FP32 ONNX model using OpenCV DNN as inference framework. The claimed accuracy is either from the source repository or paper on the same dataset, which needs to be specified in the first comment of the pull request.
  2. Once it is reproduced, apply the same evaluation script on FP16 and Int8-quantized models.

Take a look at the task list below for current status. Feel free to leave a comment for application or discussion before you start to contribute.

Status Task Dataset Models Notes
✅ Done in #70 Face Detection WIDERFace YuNet -
✅ Done in #72 Face Recognition LFW SFace -
❗️ Need Contribution License Plate Detection ? LPD-YuNet -
❗️ Need Contribution Object Detection COCO YOLOX & NanoDet Refer to #91
❗️ Need Contribution Text Detection ? DB -
✅ Done in #71 Text Recognition ICDAR2003 & IIIT5K CRNN (EN & CN) -
✅ Done in #69 Image Classification ImageNet PP-ResNet50 & MobileNet V1 / V2
✅ Done in #130 Human Segmentation Mini Supervisely Persons PP-HumanSeg -
❗️ Need Contribution QR Code Detection / Parsing ? WeChatQRCode -
❗️ Need Contribution Person Re-identification ? YoutuReID -
❗️ Need Contribution Palm Detection ? MP-PalmDet -
❗️ Need Contribution Hand Pose Estimation ? MP-HandPose -
❗️ Need Contribution Person Detection ? MP-PersonDet -
❗️ Need Contribution Pose Estimation ? MP-Pose -
❗️ Need Contribution Facial Expression Recognition RAF-DB FER -
❗️ Need Contribution Object Tracking ? VitTrack Could be done via #205

Metadata

Metadata

Assignees

Labels

evaluationadding tools for evaluation or bugs of eval scriptsfeatureNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions