-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
Context
OP conformance suite is validation tool checks a plugin conditions from operation implementation status perspective.
OP conformance is based on extracted from OMZ model scope operations and graphs by ov_subgraphs_dumper tool. Extracted graphs are saved as IR (OpenVINO Intermediate representation) and is stored in the public share.
OP conformance suite contains the following test types:
OpImplCheckvalidates operation support by plugin. Returntrueorfalse.Inferencecompare device inference results vs reference over extracted IR as a model. We generate synthetic tensors to get a result. Possible results arepassed,failed,hanged(interrupted by timeout),crashedandskipped(just in case-shape_modeis misaligned with graph inputs).QueryModelcheck possibility of graph execution per device. Have the same status asInferenceImportExportexports compiled model, imports it and check that models are same. Have the same status asInference
Template is a simple plugin run inference using reference implementation. It means if we will run conformance over template, we should compare TEMPLATE vs TEMPLATE results. Sometimes we have bugs inside the plugin, reference implementation or test infrastructure leads to negative test result.
What needs to be done?
Description:
We have several Reshape tests failed in operation conformance over TEMPLATE plugin. Do contributor should fix them:
conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=()
How to reproduce:
- Build the OV using -DENABLE_TESTS=ON -DENABLE_FUNCTIONAL_TESTS=ON
- Build
ov_op_conformance_teststarget - Run OP conformance to download conformance IRs and run executable file with args:
python3 /openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/run_conformance.py -d=TEMPLATE --gtest_filter="conformance_Gather/ReadIRTest.Inference/Op=Gather.8_Type=f32_Shape=dynamic_IR=1c727cc96123227a9fe6c3079a497fd64a04f273bff45b5ea56a3c0d577eca8e_Device=TEMPLATE_Config=()"
NOTE:
run_conformancelog contains all commands to runov_op_conformance_tests! You can take them to debug!
- Check the logs inside working directory and get the results as an example:
[ RUN ] conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=()
MEM_USAGE=71868KB
[ CONFORMANCE ] Influence coefficient: 2.89071e-06
[ PLUGIN ] `SubgraphBaseTest::compile_model()` is started
[ PLUGIN ] `SubgraphBaseTest::compile_model()` is finished successfully. Duration is 0.106113s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 0.00207637s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] Calculate reference in runtime
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.00844726s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 0.0104847s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] rel_threshold: 1.79769e+308
[ COMPARATION ] abs_threshold: 0.396962
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 0.00496672s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is finished successfully. Duration is 10.222s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] Calculate reference in runtime
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is started
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 12.747s
[ REFERENCE ] `SubgraphBaseTest::calculate_refs()` is finished successfully. Duration is 12.748s
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is started
[ COMPARATION ] rel_threshold: 1.79769e+308
[ COMPARATION ] abs_threshold: 0.396962
[ COMPARATION ] `ov_tensor_utils.hpp::compare()` is finished successfully. Duration is 82.1662s
[ PLUGIN ] `SubgraphBaseTest::get_plugin_outputs()` is started
/home/efode/repo/openvino/src/tests/functional/shared_test_classes/src/base/ov_subgraph.cpp:94: Failure
Exception from src/inference/src/cpp/infer_request.cpp:223:
Exception from src/core/src/runtime/allocator.cpp:69:
std::bad_alloc
[ FAILED ] conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=(), where GetParam() = ("/home/efode/repo/openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/2023.3.0-13657-d5b0f4d2d73/operation/dynamic/Add-1/f32/f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815.xml", "") (115025 ms)
- To debug the test in C++ Use the following command (just an example, refer to note in third item):
openvino/bin/intel64/Release/ov_op_conformance_tests --device=TEMPLATE --input_folders=openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/models/conformance_ir_files.lst, --report_unique_name --output_folder="openvino/src/tests/test_utils/functional_test_utils/layer_tests_summary/temp/report/parallel" --gtest_filter="conformance_Add/ReadIRTest.Inference/Op=Add.1_Type=f32_Shape=dynamic_IR=f86f86769ec214942eaf1fdcd312a29e26308676419d8fbd98fdc485c2de0815_Device=TEMPLATE_Config=()" --config_path="" --shape_mode=
Resources
- Contribution guide - start here!
- Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers
- How to link your Pull Request to an issue
- Conformance readme
- TEMPLATE plugin
- OV CONTRIBUTION GUIDE
Contact points
Ticket
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status