bug(barcode-scanning): corner points seem to be calculated wrong on android

[cst1412]

Plugin(s)

• Barcode Scanning
• Face Detection
• Face Mesh Detection
• Selfie Segmentation
• Translation

Version

7.0.0, 7.1.0

Platform(s)

• Android
• iOS
• Web

Current behavior

when trying to scan a barcode within the sight (box) with the demo app it will not recognize that the barcode is within the boundaries of the sight. On IOS this works fine.
When moving further away from the barcode the app will scan the barcode.

Expected behavior

The scanner should scan a barcode when its within the boundaries of the sight.

Reproduction

https://github.com/robingenz/capacitor-mlkit-plugin-demo

Steps to reproduce

1. clone and start the demo app on android.
2. upgrade dependencies (optional as it does not work on 7 and 7.1 for me)
3. click start scan (the first button)
4. try to scan a barcode while being "close" to the barcode
5. Move further away from the barcode -> the scan will work

screen-20250306-084735.mp4

Other information

i am currently in the process of upgrading our app from capacitor 6 to 7 and noticed that the scanner does not work well on android. When using version 6.2 (@capacitor-mlkit/barcode-scanning) the scan works fine and is very fast with version 7 it starts to have the issue described above.

Today i tried if the issue exists with the demo app that is available and it has the same issue.
When you log what the "barcodesScanned" event emits you can see that it will detect the barcode but the corner points are outside of the sight

The device i am using is a Fairphone 4 with Android 13

Capacitor doctor

💊 Capacitor Doctor 💊

Latest Dependencies:

@capacitor/cli: 7.0.1
@capacitor/core: 7.0.1
@capacitor/android: 7.0.1
@capacitor/ios: 7.0.1

Installed Dependencies:

@capacitor/cli: 7.0.0
@capacitor/core: 7.0.1
@capacitor/ios: 7.0.1
@capacitor/android: 7.0.1

[success] iOS looking great! 👌
[success] Android looking great! 👌

Before submitting

• I have read and followed the bug report guidelines.
• I have attached links to possibly related issues and discussions.
• I understand that incomplete issues (e.g. without reproduction) are closed.

[robingenz]

I can reproduce the problem, even if it is not quite as bad for me and can be improved by making the scan area a little larger under Android. But the problem should also have been present in version 6.2.0. This is the only difference between version 6.2.0 and version 7.1.0 that affects the corner points: @capacitor-mlkit/barcode-scanning@6.2.0...main#diff-f0d9a9e6295d00f3098bbf5db4813cceb28be36b0735215260c3a4c8b88a01c2R32 However, this is only the support for corner points for the readBarcodesFromImage(...) method. Feel free to debug it and create a PR with a fix.

[cst1412]

I had time to investigate this today and i managed to get it working with removing the following line.

obviously this is not a solution but i am confused because i set the resolution to 640x480 because it was the default in 6.2.
And while debugging the config has the right value but later while normalizing the corner points the resolution still is 1280x720.

when i do not set the target resolution the image size is correct size: 640x480.
The behavior does not change when i set the resolution to 1920x1080.

I will investigate this further but i am not really an expert when it comes to android development :D

[robingenz]

Interesting. Perhaps the target resolution is not taken into account when calculating the corner points. I'm curious to see if you can find a solution.

[cst1412]

Okay i got it working with the resolution it seems that "setTargetResolution" has a bug.
I upgraded all the androidx.camera packages to 1.3.0 and used the resolution selector as "setTargetResolution" was deprecated.
And now the imageSize is correct and the scan works flawlessly.

[cst1412]

But i cannot judge what this changes regarding device support and other things.
If this is an acceptable solution i can open a PR with these changes.

[robingenz]

Unfortunately, this is a breaking change since we have to bump third-party dependencies which might cause conflicts with other packages. Feel free to create a PR and we will publish it with the next major update which will be released as soon as Capacitor 8 is available. In the meantime, you can patch the plugin. Thank you for your help!

EDIT: You're fix is now available as pre-release:

npm i https://pkg.pr.new/capawesome-team/capacitor-mlkit/@capacitor-mlkit/barcode-scanning@257