image_segmentation/kotlin_npu/android_jit fail on SM8750

[istoneyou]

Hi, I have been playing around with the sample project https://github.com/google-ai-edge/litert-samples/tree/main/compiled_model_api/image_segmentation/kotlin_npu/android_jit and noticed that model compilation fails with the error:
13:52:45.328 20824-20847 litert E [dispatch_delegate.cc:122] Failed to create a dispatch delegate kernel: No usable Dispatch runtime foun
1```
3:52:43.021 20824-20847 qnn I INFO: [Qnn] Succssfully get platform info. SoC model: 69. SoC name: SM8750.
13:52:43.021 20824-20847 qnn I INFO: [Qnn] Using online SoC info. SoC name: SM8750.
13:52:43.021 20824-20847 qnn I INFO: [Qnn] Initializing QNN backend for SoC model: SM8750
13:52:43.059 20824-21369 com.google.ai.edg...mage_segmentation D vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:382: apps_std_fopen_fd done for /data/app/~~Qg7LoJ66A5xWYYLKgLdxFQ==/com.google.ai.edge.examples.image_segmentation-i59jHSPXoqGlPMXjZN1D3Q==/lib/arm64/cdsp/./libQnnHtpV79Skel.so with fopen:6us, read:0us, rpc_alloc:0us, mmap:0us fd 0x0 error_code 0x
13:52:43.062 20824-21369 com.google.ai.edg...mage_segmentation D vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:382: apps_std_fopen_fd done for /data/app/~~Qg7LoJ66A5xWYYLKgLdxFQ==/com.google.ai.edge.examples.image_segmentation-i59jHSPXoqGlPMXjZN1D3Q==/lib/arm64/./libQnnHtpV79Skel.so with fopen:8us, read:1839us, rpc_alloc:81us, mmap:336us fd 0xfb error_code
13:52:43.062 20824-21369 com.google.ai.edg...mage_segmentation I vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:1160: Successfully opened file /data/app/~~Qg7LoJ66A5xWYYLKgLdxFQ==/com.google.ai.edge.examples.image_segmentation-i59jHSPXoqGlPMXjZN1D3Q==/lib/arm64/./libQnnHtpV79Skel.so
13:52:43.104 20824-20847 com.google.ai.edg...mage_segmentation I vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1799: remote_handle64_open: opened handle 0xb400007795b58b50 (remote 0x2d3080) for file:///libQnnHtpV79Skel.so?qnn_skel_handle_invoke&_modver=1.0&_dom=cdsp on domain 3 (spawn time 34087 us, load time 46127 us), refs 1
13:52:43.117 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:286] QNN manager created
13:52:43.242 20824-20847 litert I [compiler_plugin.cc:472] Partition strategy: 0
13:52:43.242 20824-20847 litert I [model.cc:588] DCE removed 174 ops, 371 tensors
13:52:43.242 20824-20847 litert I [compiler_plugin.cc:645] Partitioned subgraph<0>, selected 175 ops, from a total of 175 ops. resulted in 1 partitions.
13:52:43.242 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:349] Starting QNN Compilation for 1 subgraphs, soc_model=(null)
13:52:43.242 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:414] Creating context handle
13:52:43.242 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:451] Context handle created
13:52:43.243 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:466] Composing graph
13:52:43.243 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:470] Entry point name: qnn_partition_0
13:52:45.292 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:476] Graph composed
13:52:45.292 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:489] Generating context binary
13:52:45.309 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/qnn_manager.cc:316] Serialized a context bin of size (bytes): 10391552
13:52:45.309 20824-20847 tflite I [third_party/odml/litert/litert/vendors/qualcomm/compiler/qnn_compiler_plugin.cc:492] Context binary 0 generated
13:52:45.328 20824-20847 litert I [compiled_model.cc:406] 1 compiler plugins were applied successfully: Qualcomm compiler plugin (ver 0.1.0)
13:52:45.328 20824-20847 litert W [compiled_model.cc:408] Plugin errs:
13:52:45.328 20824-20847 litert I [compiled_model.cc:716] JIT compilation changed model, reserializing...
13:52:45.328 20824-20847 litert I [dispatch_delegate.cc:172] Dispatch API vendor ID:
13:52:45.328 20824-20847 litert I [dispatch_delegate.cc:176] Dispatch API build ID: Qualcomm
13:52:45.328 20824-20847 litert I [dispatch_delegate.cc:181] Dispatch API version: 0.1.0
13:52:45.328 20824-20847 litert I [dispatch_delegate.cc:192] Dispatch API capabilities: -1506997480
13:52:45.328 20824-20847 litert E [dispatch_delegate.cc:107] Failed to initialize Dispatch API: Dispatch API has insufficient capabilities: -1506997480
13:52:45.328 20824-20847 tflite I Replacing 1 out of 1 node(s) with delegate (DispatchDelegate) node, yielding 1 partitions for subgraph 0.
13:52:45.328 20824-20847 litert E [dispatch_delegate.cc:122] Failed to create a dispatch delegate kernel: No usable Dispatch runtime found

[sourcelite]

Hi @istoneyou, great you are probably first community member to test LiteRT on this newest hardware - Snapdragon 8 Elite (SM8750). your logs telling the NPU compilation is success.

failure root cause: dispatch layer is the one who hands the compiled model to the NPU driver failed towards the end part.

root cause logs -

13:52:45.328 litert I [dispatch_delegate.cc:181] Dispatch API version: 0.1.0 
13:52:45.328 litert I [dispatch_delegate.cc:192] Dispatch API capabilities: -1506997480 
13:52:45.328 litert E [dispatch_delegate.cc:107] Failed to initialize Dispatch API: Dispatch API has insufficient capabilities: -1506997480

what is this dispatch mismatch - dispatch api is seen as bridge between litert core and qcomm specific drivers. Dispatch api version: 0.1.0. I will check more on this and share you info further...

[istoneyou]

Hi @istoneyou, great you are probably first community member to test LiteRT on this newest hardware - Snapdragon 8 Elite (SM8750). your logs telling the NPU compilation is success.

failure root cause: dispatch layer is the one who hands the compiled model to the NPU driver failed towards the end part.

root cause logs -

13:52:45.328 litert I [dispatch_delegate.cc:181] Dispatch API version: 0.1.0 
13:52:45.328 litert I [dispatch_delegate.cc:192] Dispatch API capabilities: -1506997480 
13:52:45.328 litert E [dispatch_delegate.cc:107] Failed to initialize Dispatch API: Dispatch API has insufficient capabilities: -1506997480

what is this dispatch mismatch - dispatch api is seen as bridge between litert core and qcomm specific drivers. Dispatch api version: 0.1.0. I will check more on this and share you info further...

[istoneyou]

Hi @sourcelite
Thank you for your timely response. I'm very excited about LiteRT's support for NPUs. I have a few questions:

From the existing documentation, LiteRT's support for Qualcomm NPU is mainly focused on flagship chips like SM8850 and SM8750. Are there plans to support other Qualcomm chips? For example, as far as I know, SM8845 and SM8850 both use the V81 DSP architecture.

For a PyTorch model that we want to run inference on mobile NPU using LiteRT JIT, what are the quantization requirements when converting it to TFLite via litert-torch? Can we use TensorFlow's dynamic quantization (quantizing only weights without quantizing activations)? Or is full integer quantization required?

[istoneyou]

Apologies, I closed this issue by mistake.