Larq Compute Engine seems incompatible with tensorflow-lite-task-vision on Android (using the latest tensorflow lite demo code)

[BenCrulis]

I would like to build an Android app using the Larq Compute Engine.
I started by following the Android tutorial of the Larq website, however I had to adapt the tutorial quite a bit since the tensorflow lite demo project that is referenced changed quite a bit since the moment the tutorial was written in the Larq documentation.

I added the quicknet.tflite model from the Larq zoo to the project, but I am getting the following error:

2024-01-26 14:17:42.513 11652-11766 helper                  org....examples.imageclassification  I  using model quicknet.tflite
2024-01-26 14:17:42.515 11652-11766 tflite                  org....examples.imageclassification  I  Created TensorFlow Lite XNNPACK delegate for CPU.
2024-01-26 14:17:42.515 11652-11766 tflite                  org....examples.imageclassification  E  Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.515 11652-11766 tflite                  org....examples.imageclassification  E  Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.515 11652-11766 tflite                  org....examples.imageclassification  E  Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.515 11652-11766 tflite                  org....examples.imageclassification  E  Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.517 11652-11766 TaskJniUtils            org....examples.imageclassification  E  Error getting native address of native library: task_vision_jni
                                                                                                    java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
                                                                                                    	at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
                                                                                                    	at java.lang.Thread.run(Thread.java:923)
2024-01-26 14:17:42.521 11652-11766 AndroidRuntime          org....examples.imageclassification  E  FATAL EXCEPTION: pool-2-thread-1
                                                                                                    Process: org.tensorflow.lite.examples.imageclassification, PID: 11652
                                                                                                    java.lang.IllegalStateException: Error getting native address of native library: task_vision_jni
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:95)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
                                                                                                    	at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
                                                                                                    	at java.lang.Thread.run(Thread.java:923)
                                                                                                    Caused by: java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
                                                                                                    	... 14 more
2024-01-26 14:17:42.522 11652-11770 helper                  org....examples.imageclassification  I  using model quicknet.tflite
2024-01-26 14:17:42.523 11652-11770 tflite                  org....examples.imageclassification  E  Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.523 11652-11770 tflite                  org....examples.imageclassification  E  Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.523 11652-11770 tflite                  org....examples.imageclassification  E  Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
2024-01-26 14:17:42.523 11652-11770 tflite                  org....examples.imageclassification  E  Node number 3 (LceQuantize) failed to prepare.
2024-01-26 14:17:42.527 11652-11770 TaskJniUtils            org....examples.imageclassification  E  Error getting native address of native library: task_vision_jni
                                                                                                    java.lang.IllegalArgumentException: Error occurred when initializing ImageClassifier: Encountered unresolved custom op: LceQuantize.
                                                                                                    See instructions: https://www.tensorflow.org/lite/guide/ops_custom
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.initJniWithModelFdAndOptions(Native Method)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.access$000(ImageClassifier.java:73)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:142)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier$1.createHandle(ImageClassifier.java:135)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils$1.createHandle(TaskJniUtils.java:70)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromLibrary(TaskJniUtils.java:91)
                                                                                                    	at org.tensorflow.lite.task.core.TaskJniUtils.createHandleFromFdAndOptions(TaskJniUtils.java:66)
                                                                                                    	at org.tensorflow.lite.task.vision.classifier.ImageClassifier.createFromFileAndOptions(ImageClassifier.java:133)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.setupImageClassifier(ImageClassifierHelper.java:172)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.ImageClassifierHelper.classify(ImageClassifierHelper.java:187)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.classifyImage(CameraFragment.java:349)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment.lambda$bindCameraUseCases$7$org-tensorflow-lite-examples-imageclassification-fragments-CameraFragment(CameraFragment.java:315)
                                                                                                    	at org.tensorflow.lite.examples.imageclassification.fragments.CameraFragment$$ExternalSyntheticLambda10.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysis.lambda$setAnalyzer$2(ImageAnalysis.java:481)
                                                                                                    	at androidx.camera.core.ImageAnalysis$$ExternalSyntheticLambda2.analyze(Unknown Source:2)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer.lambda$analyzeImage$0$androidx-camera-core-ImageAnalysisAbstractAnalyzer(ImageAnalysisAbstractAnalyzer.java:285)
                                                                                                    	at androidx.camera.core.ImageAnalysisAbstractAnalyzer$$ExternalSyntheticLambda1.run(Unknown Source:14)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
                                                                                                    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
                                                                                                    	at java.lang.Thread.run(Thread.java:923)

Second test

I tried to make another blank project just to see if I could execute an inference with a binary model using Larq at all, and I succeeded by only using the following dependencies in this other project:

implementation 'org.larq:lce-lite:0.1.000'
implementation 'org.tensorflow:tensorflow-lite:2.14.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.14.0'

I tried other versions of the tflite packages like 1.14.0 and 2.8.0 but it throwed JNI errors when creating the Interpreter object.

Returning to the tensorflow demo project and the Larq tutorial

The tensorflow demo uses the following dependencies:

implementation 'org.tensorflow:tensorflow-lite:2.14.0'
implementation 'org.tensorflow:tensorflow-lite-gpu:2.14.0'

// Tensorflow lite dependencies
implementation 'org.tensorflow:tensorflow-lite-support:0.4.0'
implementation 'org.tensorflow:tensorflow-lite-task-vision:0.4.0'

// Import the GPU delegate plugin Library for GPU inference
implementation 'org.tensorflow:tensorflow-lite-gpu:2.9.0'
implementation 'org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.0'

Note that I am not excluding the tensorflow-lite module like in the tutorial, I am using the following instead:

packagingOptions {
        pickFirst 'lib/arm64-v8a/libtensorflowlite_jni.so'
        pickFirst 'lib/x86/libtensorflowlite_jni.so'
        pickFirst 'lib/x86_64/libtensorflowlite_jni.so'
    }

My uneducated guess is that the issue is caused by tensorflow-lite-task-vision which seems to include its own native code libraries under other names, so that may explain the line Error getting native address of native library: task_vision_jni.

Conclusion

I guess either the tutorial can be updated to take into account the changes in the tensorflow demo repository, if failed to adapt the code correctly, or the Larq compute engine library needs to be changed to be compatible with the additional tflite modules they use in the official tflite demos.

Also, I am wondering if disabling XNNPack is still necessary? I couldn't find any way to disable it with the newest versions of tflite, but it seems to work fine with default options in my second test, I mean there is no error or warning at least.

[CNugteren]

Thank you for reporting the issue.

The last time we updated the Android tutorial was July 2021, see here. Apparently things have changed since then. I can have another look at it and see what has changed. However, I can't promise that I will be able to do this soon, so you'll either have wait a while, rely on the community to fix the tutorial, or for now switch to the TensorFlow Lite version that was the latest in July 2021.

[BenCrulis]

@CNugteren Thank you for looking into it. For now I am using the same libraries as in my second test but with tensorflow-lite-support added. Basically I am just throwing away all of the code relying on tensorflow-lite-task-vision and so far it seems to work with the library versions I specified.
If I find any other issue I will add it here.