[Build] Incompatible binaries error for multiple ORT and QNN version combinations on AArch64-Linux (including ORT 1.18.1 and QNN 2.22.10)

[deivard]

Describe the issue

I am trying to use QNN execution provider on a QCS8550 based dev kit but cannot get it to work. Cross compiling from a BitBake environment to arm works fine, but during runtime I get stuck on "QNN SetupBackend failed Failed to create device. Error: 1008", which indicates incompatible binaries. I have tried multiple combinations of ORT and QNN, but cannot get anything to work. The combination I really thought would work is ORT 1.18.1 and Qnn 2.22.10, but no.

Runtime config I used is QNN:soc_model: 66 QNN:htp_arch: 73 QNN:backend_path: libQnnHtp.so for a QCS8550 chip, which should be correct.

With some combination of versions, it wont build, and for some I get "hardware unavailable error". But it seems that the furthest I have gotten is the incompatible binaries error.

Thankful for any advice or proven to work combination of ORT and QNN on Linux ARM.

Urgency

Blocker for our project.

Target platform

AArch64-Linux

Build script

Custom bitbake recipe using CMake, might not be useful to share.

Error / output

Logs that might be of interest:

[I:onnxruntime:, inference_session.cc:533 TraceSessionOptions] Session Options {  execution_mode:0 execution_order:DEFAULT enable_profiling:0 optimized_model_filepath: enable_mem_pattern:1 enable_mem_reuse:1 enable_cpu_mem_arena:1 profile_file_prefix:onnxruntime_profile_ session_logid: session_log_severity_level:-1 session_log_verbosity_level:0 max_num_graph_transformation_steps:10 graph_optimization_level:2 intra_op_param:OrtThreadPoolParams { thread_pool_size: 0 auto_set_affinity: 0 allow_spinning: 1 dynamic_block_base_: 0 stack_size: 0 affinity_str:  set_denormal_as_zero: 0 } inter_op_param:OrtThreadPoolParams { thread_pool_size: 0 auto_set_affinity: 0 allow_spinning: 1 dynamic_block_base_: 0 stack_size: 0 affinity_str:  set_denormal_as_zero: 0 } use_per_session_threads:0 thread_pool_allow_spinning:1 use_deterministic_compute:0 config_options: {   QNN:htp_graph_finalization_optimization_mode: 3  QNN:htp_performance_mode: high_performance  QNN:enable_htp_fp16_precision: 1  QNN:soc_model: 66  QNN:htp_arch: 73  QNN:backend_path: libQnnHtp.so  kOrtSessionOptionsConfigAllowInterOpSpinning: 0  kOrtSessionOptionsConfigAllowIntraOpSpinning: 0 } } 
...
...
...
[I:onnxruntime:AI, qnn_backend_manager.cc:133 LoadBackend] Found valid interface, version: 2.15.0 backend provider name: HTP_QTI_AISW backend id: 6
[I:onnxruntime:AI, qnn_backend_manager.cc:361 CreateDevice] Create device.                                                                                                                         
[E:onnxruntime:, qnn_execution_provider.cc:513 GetCapability] QNN SetupBackend failed Failed to create device. Error: 1008

Visual Studio Version

No response

GCC / Compiler Version

No response

[deivard]

If any more information is needed, please let me know!

[TheBetterSolution]

If any more information is needed, please let me know!

Hi @deivard ,
I suggest you list all the dynamic libs loaded and then check whether the versions of the libs are matched, you may do the following steps to get paths of the libs:
lsof -n | grep '.so'

[deivard]

Hey @TheBetterSolution,

Thank you for your suggestion. I am working on creating a sample application that doesn't exit immediately so I can run lsof.

For now, based on the logging output, it seems that the (should be) correct library is loaded. As you can see in the attached image, QNN 2.22.10 is loaded, which should be the correct version, and it is the version that our ONNX runtime was compiled with. However, I am unsure about the libQnnHtp.so interface version 2.15.0 that is being mentioned in the output, but since it says it is a "valid interface", I can only assume that it is correct.

Here are some additional logs that I interpret as the HTP device was created successfully but it fails on DSP creation, which is strange since I have not specified DSP usage.

2000-01-01 02:17:45.742688788 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:74 GetQnnInterfaceProvider] libQnnHtp.so interface version: 2.15.0
2000-01-01 02:17:45.742698944 [I:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:133 LoadBackend] Found valid interface, version: 2.15.0 backend provider name: HTP_QTI_AISW backend id: 6
2000-01-01 02:17:45.742702798 [V:onnxruntime:, qnn_backend_manager.cc:690 SetupBackend] LoadBackend succeed.
2000-01-01 02:17:45.742718788 [V:onnxruntime:, qnn_backend_manager.cc:697 SetupBackend] Backend build version: v2.22.10.240618114939_92936
2000-01-01 02:17:45.742725246 [V:onnxruntime:, qnn_backend_manager.cc:258 InitializeQnnLog] Set Qnn log level: 5
2000-01-01 02:17:45.742731131 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnLog_create started.

2000-01-01 02:17:45.742747173 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <W> Initializing HtpProvider

2000-01-01 02:17:45.742754308 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Creating default router

2000-01-01 02:17:45.742762538 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> RouterFastRPC create

2000-01-01 02:17:45.742838579 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: Initializing the router

2000-01-01 02:17:45.742957017 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> exit with 0

2000-01-01 02:17:45.742966600 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> exit with 0

qnn_sample I  1796  1796 BufferAllocator.cpp:90] Using DMA-BUF heap named: system
2000-01-01 02:17:45.743684933 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> First connection to QNN stub established!

2000-01-01 02:17:45.743698267 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Loading remote funcs

2000-01-01 02:17:45.743703423 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Async property not supported. Skipping setup async threads

2000-01-01 02:17:45.743713163 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Detected Snapdragon SOC QCS8550 with 2 SOCs

2000-01-01 02:17:45.743724777 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Allocating PlatformInfo struct size 120

2000-01-01 02:17:45.743728788 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: Initializing the log registry

2000-01-01 02:17:45.743732850 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: Initializing the graph registry

2000-01-01 02:17:45.743740923 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: Initializing the context registry

2000-01-01 02:17:45.743744361 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: Initializing the device registry

2000-01-01 02:17:45.743748006 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: setting isExitCalled to false

2000-01-01 02:17:45.743755975 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: initializing mem registry

2000-01-01 02:17:45.743760038 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: initializing mmap registry

2000-01-01 02:17:45.743770454 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: initializing graphId registry

2000-01-01 02:17:45.743777746 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: initializing debugFactory

2000-01-01 02:17:45.743793683 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: constructing bundle

2000-01-01 02:17:45.743806183 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP: initialization completed succesfully

2000-01-01 02:17:45.743820038 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> Applying log level 5 to 0 devices

2000-01-01 02:17:45.743827798 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> RouterFastRPC initializeNativeOpValidatorLogs

2000-01-01 02:17:45.743831652 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <W> Function not called, PrepareLib isn't loaded!

2000-01-01 02:17:45.743848944 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> HTP Prepare backend not loaded

2000-01-01 02:17:45.743857173 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnLog_create exit.

2000-01-01 02:17:45.743866183 [V:onnxruntime:, qnn_backend_manager.cc:701 SetupBackend] SetLogger succeed.
2000-01-01 02:17:45.743870715 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnBackend_create started. backend = 0xa36d1d78

2000-01-01 02:17:45.743896652 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Backend handle created: 1

2000-01-01 02:17:45.743903423 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnBackend_create done successfully. backend = 0xa36d1d78

2000-01-01 02:17:45.743907225 [V:onnxruntime:, qnn_backend_manager.cc:704 SetupBackend] InitializeBackend succeed.
2000-01-01 02:17:45.743913579 [I:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:407 CreateDevice] Create device.
2000-01-01 02:17:45.743920767 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnDevice_create started

2000-01-01 02:17:45.743929933 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Create device with id 0x1

2000-01-01 02:17:45.743937381 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Validating device config 0x7ff1c493b8

2000-01-01 02:17:45.743944204 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Setting device config 0x7ff1c493b8 via router

2000-01-01 02:17:45.743952121 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <W> Specified config SOC, ignoring on real target

2000-01-01 02:17:45.743959933 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Validating device config 0x7ff1c493c8

2000-01-01 02:17:45.743968111 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Setting device config 0x7ff1c493c8 via router

2000-01-01 02:17:45.743978111 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <W> HTP arch will be deprecated, please set SoC id instead.

2000-01-01 02:17:45.743985975 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Config not passed. Loading default config

2000-01-01 02:17:45.743989413 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Setting default value for unsigned PD usage

2000-01-01 02:17:45.744000350 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> skel file path file:///libQnnHtpV73Skel.so?qnn_skel_handle_invoke&_modver=1.0&_dom=cdsp

2000-01-01 02:17:45.744004413 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <W> sg_stubPtr is not null, skip loadRemoteSymbols


/prj/qct/webtech_hyd18/mlg_user_admin/qaisw_source_repo/qaisw_repo_point_release/snpe_src/avante-tools/prebuilt/dsp/hexagon-sdk-5.4.0/ipc/fastrpc/rpcmem/src/rpcmem_android.c:38:dummy call to rpcmem_init, rpcmem APIs will be used from libxdsprpc
2000-01-01 02:17:45.744013892 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> exits with 2, successfully initialized rpc memory

2000-01-01 02:17:45.744019621 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> Successful rpcMemInit

2000-01-01 02:17:45.744023996 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> rpcMemoryAlloc 8 isInit 1

2000-01-01 02:17:45.744149204 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> rpcMemoryAlloc 136 isInit 1

2000-01-01 02:17:45.769194569 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> DspTransport.openSession qnn_open failed, 0x80000406

2000-01-01 02:17:45.769219048 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> IDspTransport: Unable to load lib 0x80000406

2000-01-01 02:17:45.769224986 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> DspTransport failed,cannot open session, error 0x00000009

2000-01-01 02:17:45.769231288 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Unable to load Skel Library. transportStatus: 9

2000-01-01 02:17:45.769243423 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to retrieve skel build id: err: 1008

2000-01-01 02:17:45.769298475 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to create transport for device, error: 1008

2000-01-01 02:17:45.769311079 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to load skel, error: 1008

2000-01-01 02:17:45.769316027 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> exits device initialization with  1008

2000-01-01 02:17:45.769320923 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Transport layer setup failed: 1008

2000-01-01 02:17:45.769331861 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to parse default platform info: 1008

2000-01-01 02:17:45.769336444 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to load default platform info: 1008

2000-01-01 02:17:45.769348475 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to parse platform config: 1008

2000-01-01 02:17:45.769361288 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <I> QnnDevice_create done. status 0x3f0

2000-01-01 02:17:45.769377850 [E:onnxruntime:, qnn_execution_provider.cc:580 GetCapability] QNN SetupBackend failed Failed to create device. Error: QNN_COMMON_ERROR_INCOMPATIBLE_BINARIES: Loaded libraries are of incompatible versions

[deivard]

After creating the sample application and running lsof -n | grep libQnn, I get the following output:

I am not sure if I am getting the errors because of the following line in the output from my previous comment:

2000-01-01 02:17:45.744000350 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> skel file path file:///libQnnHtpV73Skel.so?qnn_skel_handle_invoke&_modver=1.0&_dom=cdsp
...
...
2000-01-01 02:17:45.769231288 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Unable to load Skel Library. transportStatus: 9
2000-01-01 02:17:45.769243423 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to retrieve skel build id: err: 1008
2000-01-01 02:17:45.769298475 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to create transport for device, error: 1008
2000-01-01 02:17:45.769311079 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to load skel, error: 1008
2000-01-01 02:17:45.769316027 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <V> exits device initialization with  1008
2000-01-01 02:17:45.769320923 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Transport layer setup failed: 1008
2000-01-01 02:17:45.769331861 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to parse default platform info: 1008
2000-01-01 02:17:45.769336444 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to load default platform info: 1008
2000-01-01 02:17:45.769348475 [V:onnxruntime:QNNSampleEnv, qnn_backend_manager.cc:249 QnnLogging] QnnDsp <E> Failed to parse platform config: 1008

I believe that err: 1008 means incompatible binaries, but why they are incompatible is the real question. Is it because of the transportStatus: 9 error? If so, what could that mean? Perhaps the library isn't found by the application and that results in a transportStatus: 9, or maybe it is a version of the libQnnHtpV73Skel.so that is incompatible with the QCS8550 chip.

Regardless, I am starting to suspect that this is an issue that is more closely linked to the Qualcomm hardware than ORT and that the binary incompatibility errors aren't talking about the ONNX runtime and QNN libraries, but the QNN HTP library and the hardware. The incompatible binaries error seems to be propagated from the QnnDsp library trying to load libQnnHtpV73Skel.so.

[deivard]

For anyone running into this in the future: I was able to resolve this error by setting the ADSP_LIBRARY_PATH environment variable to the directory containing libQnnHtpV73Skel.so on the target device before running.

More information about this error can be found here.

All credits to this solution goes to the Discord user Eric on the Qualcomm Developers Discord channel.