Skip to content

Commit f02a640

Browse files
qti-hungjuiwqti-hungjuiw
authored
[QNN-EP] Enable verbose and artifacts saving in onnxruntime_provider_test.exe (#26396)
### Description <!-- Describe your changes. --> - The change allows users to better debug unit tests by adding the following environment variables: - `QNN_DUMP_ONNX`: Dump input onnx model - `QNN_DUMP_JSON`: Dump json qnn graph with provider_option `dump_json_qnn_graph` - `QNN_DUMP_DLC`: Dump dlc with provider_option `qnn_ir_backend_path` - `QNN_VERBOSE`: Use the log level `ORT_LOGGING_LEVEL_VERBOSE` - Developers can use the environment variables above to save the artifacts of QNN-EP testcases to a directory named with `<TestSuite>_<TestName>` ``` . ├── QnnCPUBackendTests_BatchNorm2D_fp32 # RunQnnModelTest │ ├── dumped_f32_model.onnx # float32 ONNX model │ ├── QNNExecutionProvider_QNN_XXXX_X_X.dlc │ └── QNNExecutionProvider_QNN_XXXX_X_X.json ├── QnnHTPBackendTests_BatchNorm_FP16 # TestFp16ModelAccuracy │ ├── dumped_f16_model.onnx # float16 ONNX model │ ├── dumped_f32_model.onnx # float32 ONNX model │ ├── QNNExecutionProvider_QNN_XXXX_X_X.dlc │ └── QNNExecutionProvider_QNN_XXXX_X_X.json └── QnnHTPBackendTests_BatchNorm2D_U8U8S32 # TestQDQModelAccuracy ├── dumped_f32_model.onnx # float32 ONNX model ├── dumped_qdq_model.onnx # QDQ ONNX model ├── QNNExecutionProvider_QNN_XXXX_X_X.dlc └── QNNExecutionProvider_QNN_XXXX_X_X.json # All artifact files are placed under the current working directory from which the test binary is invoked. ``` ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> - The Json qnn graph/dlc are helpful for backend to debug performance/accuracy issues - By comparing the onnx and Json qnn graph/dlc, we can locate the issue about graph manipulation.
1 parent 55bfa30 commit f02a640

File tree

6 files changed

+278
-8
lines changed

6 files changed

+278
-8
lines changed

‎onnxruntime/core/providers/qnn/builder/opbuilder/base_op_builder.cc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ Status BaseOpBuilder::ProcessDataTypes(QnnModelWrapper& qnn_model_wrapper,
7676
return CheckHtpDataTypes(input_qnn_dtypes, output_qnn_dtypes);
7777
} else if (IsGpuBackend(qnn_model_wrapper.GetQnnBackendType())) {
7878
return CheckGpuDataTypes(input_qnn_dtypes, output_qnn_dtypes);
79+
} else if (IsIrBackend(qnn_model_wrapper.GetQnnBackendType())) {
80+
// TODO: CheckIrDataTypes
81+
return Status::OK();
7982
}
8083
return ORT_MAKE_STATUS(ONNXRUNTIME, FAIL, "Only support backend: CPU, HTP and GPU");
8184
}

‎onnxruntime/core/providers/qnn/builder/qnn_def.cc‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -574,6 +574,10 @@ bool QnnOpConfigWrapper::CreateQnnGraphOp(const QNN_INTERFACE_VER_TYPE& qnn_inte
574574
return true;
575575
}
576576

577+
bool IsIrBackend(QnnBackendType backend_type) {
578+
return backend_type == QnnBackendType::SERIALIZER;
579+
}
580+
577581
bool IsNpuBackend(QnnBackendType backend_type) {
578582
return backend_type == QnnBackendType::HTP || backend_type == QnnBackendType::DSP;
579583
}

‎onnxruntime/core/providers/qnn/builder/qnn_def.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@ enum class QnnBackendType : uint8_t {
9696
SERIALIZER,
9797
};
9898

99+
bool IsIrBackend(QnnBackendType backend_type);
100+
99101
bool IsCpuBackend(QnnBackendType backend_type);
100102

101103
bool IsNpuBackend(QnnBackendType backend_type);

‎onnxruntime/test/providers/qnn/README.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# ONNX Runtime QNN Execution Provider Tests
2+
## Overview
3+
1. The `onnxruntime/test/providers/qnn` directory contains integration tests for the Qualcomm Neural Network (QNN) execution provider.
4+
2. Most testcases run an ONNX model through the QNN-EP, then verifies the inference result against the one on CPU-EP
5+
6+
## Building the Tests
7+
The tests are built as part of the regular ONNX Runtime build. After a successful build you will have an executable named
8+
- onnxruntime_provider_test.exe (Windows)
9+
- onnxruntime_provider_test (Linux/macOS)
10+
11+
## Running the Tests
12+
1. QNN supports several backends. You can use the standard Google‑Test syntax for filtering:
13+
- `onnxruntime_provider_test.exe --gtest_filter=QnnCPUBackendTests.*`
14+
- `onnxruntime_provider_test.exe --gtest_filter=QnnHTPBackendTests.*`
15+
- `onnxruntime_provider_test.exe --gtest_filter=QnnGPUBackendTests.*`
16+
- `onnxruntime_provider_test.exe --gtest_filter=QnnIRBackendTests.*`
17+
2. Saving Test Artifacts
18+
- For debugging it is often helpful to keep the intermediate files that the tests generate. The following environment
19+
variables are recognized by the test binary:
20+
- `QNN_DUMP_ONNX`: Saves the input ONNX model used for the test
21+
- `QNN_DUMP_JSON`: Save json qnn graph with provider_option `dump_json_qnn_graph`
22+
- `QNN_DUMP_DLC`: Saves the compiled QNN DLC file by specifying the provider_option `backend_path` to `QnnIr.dll`
23+
- The artifacts will be saved to a directory named with `<TestSuite>_<TestName>`
24+
```
25+
.
26+
├── QnnCPUBackendTests_BatchNorm2D_fp32 # RunQnnModelTest
27+
│ ├── dumped_f32_model.onnx # float32 ONNX model
28+
│ ├── QNNExecutionProvider_QNN_XXXX_X_X.dlc
29+
│ └── QNNExecutionProvider_QNN_XXXX_X_X.json
30+
├── QnnHTPBackendTests_BatchNorm_FP16 # TestFp16ModelAccuracy
31+
│ ├── dumped_f16_model.onnx # float16 ONNX model
32+
│ ├── dumped_f32_model.onnx # float32 ONNX model
33+
│ ├── QNNExecutionProvider_QNN_XXXX_X_X.dlc
34+
│ └── QNNExecutionProvider_QNN_XXXX_X_X.json
35+
└── QnnHTPBackendTests_BatchNorm2D_U8U8S32 # TestQDQModelAccuracy
36+
├── dumped_f32_model.onnx # float32 ONNX model
37+
├── dumped_qdq_model.onnx # QDQ ONNX model
38+
├── QNNExecutionProvider_QNN_XXXX_X_X.dlc
39+
└── QNNExecutionProvider_QNN_XXXX_X_X.json
40+
41+
# All artifact files are placed under the current working directory from which the test binary is invoked.
42+
```
43+
3. Verbose
44+
- `QNN_VERBOSE`: Sets the ONNX Runtime log level to `ORT_LOGGING_LEVEL_VERBOSE`
45+
46+
4. You can enable any combination of these environment variables, for example:
47+
- On Linux/macOS
48+
```bash
49+
export QNN_DUMP_ONNX=1
50+
export QNN_DUMP_JSON=1
51+
export QNN_DUMP_DLC=1
52+
export QNN_VERBOSE=1
53+
```
54+
- On Windows
55+
```cmd
56+
set QNN_DUMP_ONNX=1
57+
set QNN_DUMP_JSON=1
58+
set QNN_DUMP_DLC=1
59+
set QNN_VERBOSE=1
60+
```
61+
```ps1
62+
$Env:QNN_DUMP_ONNX = "1"
63+
$Env:QNN_DUMP_JSON = "1"
64+
$Env:QNN_DUMP_DLC = "1"
65+
$Env:QNN_VERBOSE = "1"
66+
```
67+
68+
# Note
69+
- An issue on QNN backends can prevent the test artifacts from being successfully saved.
70+
- The `onnxruntime_provider_test.exe` does not automatically delete the artifact directories, so you may want to prune them after a debugging session.

‎onnxruntime/test/providers/qnn/qnn_test_utils.cc‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,12 @@ void RunQnnModelTest(const GetTestModelFn& build_test_case, ProviderOptions prov
101101
int opset_version, ExpectedEPNodeAssignment expected_ep_assignment,
102102
float fp32_abs_err, logging::Severity log_severity, bool verify_outputs,
103103
std::function<void(const Graph&)>* ep_graph_checker) {
104+
std::filesystem::path output_dir;
105+
if (QNNTestEnvironment::GetInstance().dump_onnx() ||
106+
QNNTestEnvironment::GetInstance().dump_json() ||
107+
QNNTestEnvironment::GetInstance().dump_dlc()) {
108+
output_dir = QNNTestEnvironment::GetInstance().CreateTestcaseDirs();
109+
}
104110
EPVerificationParams verification_params;
105111
verification_params.ep_node_assignment = expected_ep_assignment;
106112
verification_params.fp32_abs_err = fp32_abs_err;
@@ -110,6 +116,10 @@ void RunQnnModelTest(const GetTestModelFn& build_test_case, ProviderOptions prov
110116

111117
auto& logging_manager = DefaultLoggingManager();
112118
logging_manager.SetDefaultLoggerSeverity(log_severity);
119+
if (QNNTestEnvironment::GetInstance().verbose()) {
120+
logging_manager.RemoveSink(logging::SinkType::EtwSink);
121+
logging_manager.SetDefaultLoggerSeverity(logging::Severity::kVERBOSE);
122+
}
113123

114124
onnxruntime::Model model("QNN_EP_TestModel", false, ModelMetaData(), PathString(),
115125
IOnnxRuntimeOpSchemaRegistryList(), domain_to_version, {},
@@ -123,7 +133,27 @@ void RunQnnModelTest(const GetTestModelFn& build_test_case, ProviderOptions prov
123133
// Serialize the model to a string.
124134
std::string model_data;
125135
model.ToProto().SerializeToString(&model_data);
136+
137+
if (QNNTestEnvironment::GetInstance().dump_onnx()) {
138+
auto dump_path = output_dir / ToPathString("dumped_f32_model.onnx");
139+
LOGS(logging_manager.DefaultLogger(), VERBOSE) << "Save onnx model at: " << dump_path;
140+
ASSERT_STATUS_OK(onnxruntime::Model::Save(model, dump_path));
141+
}
142+
126143
TryEnableQNNSaver(provider_options);
144+
if (QNNTestEnvironment::GetInstance().dump_dlc()) {
145+
provider_options["dump_qnn_ir_dlc"] = "1";
146+
provider_options["dump_qnn_ir_dlc_dir"] = output_dir.string();
147+
#if defined(_WIN32)
148+
provider_options["qnn_ir_backend_path"] = "QnnIr.dll";
149+
#else
150+
provider_options["qnn_ir_backend_path"] = "libQnnIr.so";
151+
#endif // defined(_WIN32)
152+
}
153+
if (QNNTestEnvironment::GetInstance().dump_json()) {
154+
provider_options["dump_json_qnn_graph"] = "1";
155+
provider_options["json_qnn_graph_dir"] = output_dir.string();
156+
}
127157
RunAndVerifyOutputsWithEP(AsByteSpan(model_data.data(), model_data.size()), "QNN_EP_TestLogID",
128158
QnnExecutionProviderWithOptions(provider_options),
129159
helper.feeds_, verification_params,
@@ -134,11 +164,21 @@ void RunQnnModelTestHTPNoVerify(const GetTestModelFn& build_test_case, ProviderO
134164
int opset_version, ExpectedEPNodeAssignment expected_ep_assignment,
135165
logging::Severity log_severity,
136166
std::function<void(const Graph&)>* ep_graph_checker) {
167+
std::filesystem::path output_dir;
168+
if (QNNTestEnvironment::GetInstance().dump_onnx() ||
169+
QNNTestEnvironment::GetInstance().dump_dlc() ||
170+
QNNTestEnvironment::GetInstance().dump_json()) {
171+
output_dir = QNNTestEnvironment::GetInstance().CreateTestcaseDirs();
172+
}
137173
// Add kMSDomain to cover contrib op like Gelu
138174
const std::unordered_map<std::string, int> domain_to_version = {{"", opset_version}, {kMSDomain, 1}};
139175

140176
auto& logging_manager = DefaultLoggingManager();
141177
logging_manager.SetDefaultLoggerSeverity(log_severity);
178+
if (QNNTestEnvironment::GetInstance().verbose()) {
179+
logging_manager.RemoveSink(logging::SinkType::EtwSink);
180+
logging_manager.SetDefaultLoggerSeverity(logging::Severity::kVERBOSE);
181+
}
142182

143183
onnxruntime::Model model("QNN_EP_TestModel", false, ModelMetaData(), PathString(),
144184
IOnnxRuntimeOpSchemaRegistryList(), domain_to_version, {},
@@ -152,7 +192,27 @@ void RunQnnModelTestHTPNoVerify(const GetTestModelFn& build_test_case, ProviderO
152192
// Serialize the model to a string.
153193
std::string model_data;
154194
model.ToProto().SerializeToString(&model_data);
195+
196+
if (QNNTestEnvironment::GetInstance().dump_onnx()) {
197+
auto dump_path = output_dir / ToPathString("dumped_f32_model.onnx");
198+
LOGS(logging_manager.DefaultLogger(), VERBOSE) << "Save onnx model at: " << dump_path;
199+
ASSERT_STATUS_OK(onnxruntime::Model::Save(model, dump_path));
200+
}
201+
155202
TryEnableQNNSaver(provider_options);
203+
if (QNNTestEnvironment::GetInstance().dump_dlc()) {
204+
provider_options["dump_qnn_ir_dlc"] = "1";
205+
provider_options["dump_qnn_ir_dlc_dir"] = output_dir.string();
206+
#if defined(_WIN32)
207+
provider_options["qnn_ir_backend_path"] = "QnnIr.dll";
208+
#else
209+
provider_options["qnn_ir_backend_path"] = "libQnnIr.so";
210+
#endif // defined(_WIN32)
211+
}
212+
if (QNNTestEnvironment::GetInstance().dump_json()) {
213+
provider_options["dump_json_qnn_graph"] = "1";
214+
provider_options["json_qnn_graph_dir"] = output_dir.string();
215+
}
156216

157217
SessionOptions so;
158218
so.session_logid = "QNN_EP_TestLogID";

0 commit comments

Comments
 (0)