This issue tracks subtest failures that have been observed in the WebNN conformance suite for WinML's DMLExecutionProvider. Report attached for reference. Ideally, the suite would pass cleanly, so the subtests here should be investigated and fixed where possible.
To reproduce, on a compatible AMD device, run the wpt suite using the webnn-test tool. I used Chrome Canary for my repro.
The following arguments are passed to the command line:
--webnn-ort-ep-device=DmlExecutionProvider,0x1002,0x150e --webnn-ort-logging-level=VERBOSE
Test Results Link
Summary of Failures by Category
| Category |
Issue Type |
Affected Tests |
| Data Type Support |
dequantizeLinear doesn't support int32 |
dequantizeLinear, qdq_subgraph |
| Data Type Support |
gatherElements/gatherND don't support uint32 indices |
gatherElements, gatherND |
| Session Creation |
Failed to create session errors |
batchNormalization (1D), matmul (5D), quantizeLinear, prelu (int64), relu (int64), dequantizeLinear (int4) |
| Precision/ULP |
Float16/Float32 precision exceeds tolerance |
batchNormalization, l2Pool2d, layer_normalization, quantizeLinear, subgraph |
| Context Lost |
Context is lost errors |
quantizeLinear, qdq_subgraph |
| API Gap |
graph.devices is empty |
graph_devices |
Detailed Failing Tests
1. batch_normalization_constant (2 failing / 3 total)
| Subtest |
Error |
[optional] batchNormalization float32 2D constant tensors default options |
ULP distance 12 (expected ≤6) |
[optional] batchNormalization float16 2D constant tensors default options |
ULP distance 7 (expected ≤6) |
2. batch_normalization (8 failing / 25 total)
| Subtest |
Error |
[optional] batchNormalization float32 1D tensor options.axis=0 |
Failed to create session |
[optional] batchNormalization float16 1D tensor options.axis=0 |
Failed to create session |
[optional] batchNormalization float16 2D tensor (mean and variance are non-constant) default options |
ULP distance 7 (expected ≤6) |
[optional] batchNormalization float16 2D tensor default options |
ULP distance 7 (expected ≤6) |
[required] batchNormalization float16 4D tensor default options |
ULP distance 8 (expected ≤6) |
[required] batchNormalization float16 4D NCHW tensor options.axis=1 |
ULP distance 8 (expected ≤6) |
[required] batchNormalization float16 4D NHWC tensor options.axis=3 |
ULP distance 8 (expected ≤6) |
[required] batchNormalization float16 4D NCHW tensor options.epsilon |
ULP distance 8 (expected ≤6) |
3. dequantizeLinear (12 failing / 33 total)
| Subtest |
Error |
[required] dequantizeLinear int8 4D tensor broadcasting float32 4D scale and int8 4D zeroPoint |
ULP distance 33982206 (expected ≤1) |
[optional] dequantizeLinear uint4 1D tensor of odd size with float32 1D scale |
Failed to create session |
[optional] dequantizeLinear uint4 4D tensor with broadcasting float32 4D scale and uint4 4D zeroPoint |
Failed to create session |
[optional] dequantizeLinear int4 1D tensor of odd size with float32 1D scale |
Failed to create session |
[optional] dequantizeLinear int32 1D tensor with float32 1D scale |
Unsupported data type int32 (must be int8, uint8, int4, uint4) |
[required] dequantizeLinear with float32 3D scale as an intermediate node |
Unsupported data type int32 |
[optional] dequantizeLinear int8 4D tensor broadcasting float16 4D scale and int8 4D zeroPoint |
ULP distance 4149 (expected ≤1) |
[optional] dequantizeLinear uint4 1D tensor of odd size with float16 1D scale |
Failed to create session |
[optional] dequantizeLinear uint4 4D tensor with broadcasting float16 4D scale and uint4 4D zeroPoint |
Failed to create session |
[optional] dequantizeLinear int4 1D tensor of odd size with float16 1D scale |
Failed to create session |
[optional] dequantizeLinear int32 1D tensor with float16 1D scale |
Unsupported data type int32 |
[optional] dequantizeLinear with float16 3D scale as an intermediate node |
Unsupported data type int32 |
4. gatherElements (2 failing / 12 total)
| Subtest |
Error |
[optional] gatherElements float32 2D input and uint32 indices options.axis=1 |
Unsupported data type uint32 for indices (must be int32, int64) |
[optional] gatherElements float16 2D input and uint32 indices options.axis=1 |
Unsupported data type uint32 for indices |
5. gatherND (2 failing / 17 total)
| Subtest |
Error |
[optional] gatherND float32 4D input and 1D uint32 indices |
Unsupported data type uint32 for indices (must be int32, int64) |
[optional] gatherND float16 4D input and 1D uint32 indices |
Unsupported data type uint32 for indices |
6. graph_devices (1 failing / 1 total)
| Subtest |
Error |
test graph.devices |
graph.devices is empty (expected > 0) |
7. l2Pool2d (1 failing / 29 total)
| Subtest |
Error |
[optional] l2Pool2d float16 4D tensor options.padding |
ULP distance 8060 (expected ≤27) - actual: Infinity, expected: 289 |
8. layer_normalization (1 failing / 26 total)
| Subtest |
Error |
[required] layerNormalization float32 4D tensor all options |
ULP distance 35 (expected ≤14) |
9. matmul (6 failing / 23 total)
| Subtest |
Error |
[required] matmul float32 5D and 5D tensors, broadcast the two leftmost dimensions of inputB |
Failed to create session |
[required] matmul float32 5D and 5D tensors, broadcast the leftmost dimensions of inputB |
Failed to create session |
[required] matmul float32 5D and 2D tensors |
Failed to create session |
[required] matmul float16 5D and 5D tensors, broadcast the two leftmost dimensions of inputB |
Failed to create session |
[required] matmul float16 5D and 5D tensors, broadcast the leftmost dimensions of inputB |
Failed to create session |
[required] matmul float16 5D and 2D tensors |
Failed to create session |
10. prelu (1 failing / 28 total)
| Subtest |
Error |
[optional] prelu int64 2D constant tensors |
Failed to create session |
11. quantizeLinear (6 failing / 29 total)
| Subtest |
Error |
[required] quantizeLinear float32 4D tensor broadcasting scale and zeroPoint |
ULP distance 16 (expected ≤1) |
[optional] quantizeLinear float32 2D tensor with int4 zeroPoint which has even size |
Failed to create session |
[optional] quantizeLinear float32 tensor with uint4 zeroPoint which has odd size |
Failed to create session |
[optional] quantizeLinear float16 2D tensor broadcasting zeroPoint and scale |
Context is lost |
[optional] quantizeLinear float16 4D tensor broadcasting scale and zeroPoint |
ULP distance 16 (expected ≤1) |
[optional] quantizeLinear float16 2D tensor with int4 zeroPoint which has even size |
Failed to create session |
[optional] quantizeLinear float16 tensor with uint4 zeroPoint which has odd size |
Failed to create session |
12. qdq_subgraph (9 failing / 42 total)
| Subtest |
Error |
[optional] quantized conv2d |
Unsupported data type int32 for dequantizeLinear |
[optional] dequantizeLinear -> conv2d -> clamp -> quantizeLinear |
Unsupported data type int32 for dequantizeLinear |
[optional] dequantizeLinear -> conv2d -> relu -> quantizeLinear |
Unsupported data type int32 for dequantizeLinear |
[optional] quantized convTranspose2d |
Unsupported data type int32 for dequantizeLinear |
[optional] quantized conv2d with padding |
Unsupported data type int32 for dequantizeLinear |
[optional] quantized gemm with bias |
Unsupported data type int32 for dequantizeLinear |
[required] quantized leaky relu |
Context is lost |
[required] quantized averagePool2d |
Failed to create session |
[required] quantized softmax |
Context is lost |
13. relu (1 failing / 18 total)
| Subtest |
Error |
[optional] relu int64 4D tensor |
Failed to create session |
14. subgraph (1 failing / 49 total)
| Subtest |
Error |
[required] batchNormalization options.axis=0 + gelu |
ULP distance 38 (expected ≤24) |
