dioxus barcode scanner

[brianmay]

Hello,

Crazy idea maybe, but I would like a wasm front end written in dioxus to be able to prompt the user to scan a barcode.

Not sure where to start... Or if this is feasible.

I guess there are two distinct steps:

1. grab photo using camera.
2. detect any barcodes in that photo.

For step 2, It looks like there is on official detection API, https://developer.mozilla.org/en-US/docs/Web/API/Barcode_Detection_API

But by the looks of it browser support might be limited. Not sure if Mobile firefox on Android supports it, might be limited to Chrome on Android.

Then somebody has written a component for yew: https://gitlab.com/voltfang-public/yew-scanner

Code here: https://gitlab.com/voltfang-public/yew-scanner/-/blob/main/src/lib.rs?ref_type=heads

It looks like this grabs an image from a HtmlVideoElement and passes to the bardecoder crate for decoding (not sure I entirely understand how the HtmlVideoElement gets its image.

https://github.com/piderman314/bardecoder

But it does certain things which I am not sure of in Dioxus, e.g. grabbing references to rendered html elements.

Then there is some JavaScript code:

https://stackoverflow.com/questions/67078359/barcode-scanning-via-mobile-browser

And I see this library looks like it might have Camera support in the future, but isn't quite there yet:

https://github.com/DioxusLabs/sdk

Any suggestions on which approach I should take?

[alkeryn]

i figured this discussion may be helpful, you can call native api's.

[brianmay]

I am unclear what you intended here. Did you mean to add a link?

[alkeryn]

Oh yea sorry, i was tired and didn't realized it didn't paste
Here is the link. #3475

My current issue rn is more how to figure background tasks on mobile with dioxus.

[brianmay]

I had a fiddle with some code proof of concept code. https://stackoverflow.com/a/78089783, which uses the barcode scanner API. This has poor browser support, but there is a polyfill for it.

Amazingly it seems to work, at least on Desktop with Camera. Proof of concept code only though.

I can't think right now, but wondering:

1. Is there some way to npm install the polyfill package (see https://github.com/undecaf/barcode-detector-polyfill) and use that from within dioxus?
2. How do I return the scanned barcode from JavaScript to the wasm code? Ideally I would like to be able to set a signal. Somehow I suspect it may not be that simple. Particularly as the Javascript code is in a loop. I guess I could exit as soon as a barcode is found though maybe.

#[component]
fn Barcode() -> Element {
    use_effect(move || {
        document::eval(
            r#"
            console.log("00000");

            //WebAssembly polyfill for some browsers
            try { window['BarcodeDetector'].getSupportedFormats() }
            catch { window['BarcodeDetector'] = barcodeDetectorPolyfill.BarcodeDetectorPolyfill };
            console.log("1111");

            // Define video as the video element. You can pass the entire element to the barcode detector!
            const video = document.querySelector('video');
            console.log("2222");

            // Get a stream for the rear camera, else the front (or side?) camera, and show it in the video element.
            video.srcObject = await navigator.mediaDevices.getUserMedia({ video: { facingMode: 'environment' } });
            console.log("33333");

            // Create a BarcodeDetector for simple retail operations.
            const barcodeDetector = new BarcodeDetector({ formats: ["ean_13", "ean_8", "upc_a", "upc_e"] });
            console.log("4444");

            while(true) {
                try {
                    // Try to detect barcodes in the current video frame.
                    let barcodes = await barcodeDetector.detect(video);

                    // Continue loop if no barcode was found.
                    if (barcodes.length == 0)
                    {
                        // Scan interval 50 ms like in other barcode scanner demos.
                        // The higher the interval the longer the battery lasts.
                        await new Promise(r => setTimeout(r, 50));
                        continue;
                    }

                    console.log("xxxx", barcodes);

                    // We expect a single barcode.
                    // It's possible to compare X/Y coordinates to get the center-most one.
                    // One can also do "preferred symbology" logic here.
                    // document.getElementById("barcode").innerText = barcodes[0].rawValue;

                    // Notify user that a barcode has been found.
                    // navigator.vibrate(200);

                    // Give the user time to find another product to scan
                    await new Promise(r => setTimeout(r, 1000));
                }
                catch(err) {
                    console.log("failed", err);
                    //Wait till video is ready
                    //barcodeDetector.detect(video) might fail the first time
                    await new Promise(r => setTimeout(r, 200));
                }
            }
        "#,
        );
    });

    rsx! {
        script { src: "https://cdn.jsdelivr.net/npm/@undecaf/zbar-wasm@0.9.15/dist/index.js" }
        script { src: "https://cdn.jsdelivr.net/npm/@undecaf/barcode-detector-polyfill@0.9.21/dist/index.js" }
        h1 { "Penguins Rule"}
        video { autoplay: true, playsinline: true,  }
    }
}

[brianmay]

I managed to integrate this into my app at https://github.com/brianmay/penguin_nurse/

I used https://www.youtube.com/watch?v=ZF0IPdSVbPw for inspiration.

But I did not use esbuild, as it seemed to have problems building the library I needed. See undecaf/barcode-detector-polyfill#20

Instead I used rollup. And the the rollup.config.mjs from https://github.com/undecaf/barcode-detector-polyfill/tree/master/example-bundled which renames some references inside the code. Yes, this seems yuck. But this is required if you want to avoid using an external copy of zbar.wasm. Which isn't a huge thing, apart from a large comment it is already minified by rollup anyway.

I couldn't use this output in the document::eval, it had import statements which failed. And I couldn't work out how to configure rollup to stop this.

So I used in my top level page.

    rsx! {
        document::Script {
            type: "module",
            src: asset!("/assets/bundle.js", JsAssetOptions::new().with_minify(false)),
        }
    ...
    }

Where bundle just loads the polyfill to window['BarcodeDetector'], and does nothing else.

Then in my document::eval I can use the already loaded polyfill as before, with inline javascript code. https://github.com/brianmay/penguin_nurse/blob/bfeb630f18f862b645f7068e578d78cb11eb2aef/src/forms/barcodes.rs#L8-L65

I also had problems with the js minification of the file creating an invalid import line. As it created an import line with the absolute filesystem path, which Firefox complained loudly. So (as above) I disabled that.

Uncaught (in promise) TypeError: URL constructor: /home/brian/tree/personal/penguin_nurse/assets/bundle.js is not a valid URL.

I think this was caused by the minified file containing:

const importMeta={url:"/home/brian/tree/personal/penguin_nurse/assets/bundle.js" ...

Wondering if this is a bug.

Then I had the problem then that barcode-detector-polyfill really wants to be able to load /assets/zbar.wasm

I can include it using:

let _zbar = asset!("/assets/zbar.wasm");

but dioxus wants to serve it as /assets/zbar-6a921bd2bb9606f1.wasm, and I couldn't work out how to do this with barcode-detector-polyfill.

So I had to rename that inside the js also: https://github.com/brianmay/penguin_nurse/blob/bfeb630f18f862b645f7068e578d78cb11eb2aef/rollup.config.mjs#L24

Not sure I like this, it will break if the file changes. But best I can think of right now.

[Lebei2046]

QR code scanning is a very popular function for mobile apps. I'm also looking for a solution for Dioxus. In React, @yudiel/react-qr-scanner works. But needs hard work to port.

[brianmay]

My solution using zbar seems to work. Can provide up-to-date source references if you want. I have a working Dioxus component. Just same limitations (which may or may not be zbar's fault):

• Sometimes won't focus on barcode. I have to switch to camera app, focus, and then come back. This might be a problem with my JavaScript/html code that connects to the camera, I might have missed something required to enable auto-focus.
• Seems to have high error rate scanning barcodes. i.e. the data returned looks nothing like the barcode, and I have to keep retrying until I get the correct value. I noticed the same thing on a bar code scanning app too. Haven't investigated, I guess problem could be that error gets past check sum checking, or check sum checking isn't getting done.
• Haven't found a good way to tell the browser that we have finished with the camera. So browser won't let user use the light with app activate, because it thinks the app is still using the camera, when it isn't.

[w1n5t0n99]

If you're looking for a paid solution this company put out a Dioxus example.
https://scanbot.io/techblog/dioxus-barcode-scanner-rust-tutorial/

[Ekcum]

use dioxus::logger::tracing;
use dioxus::prelude::*;
use dioxus_primitives::accordion::{Accordion, AccordionContent, AccordionItem, AccordionTrigger};

#[derive(PartialEq)]
enum ScanningStatus {
    Scanning = 0,
    ReadyToScan = 1,
    UnableToScan = 2,
}

impl ScanningStatus {
    pub fn to_string(&self) -> String {
        match self {
            ScanningStatus::Scanning => String::from("Scanning"),
            ScanningStatus::ReadyToScan => String::from("ReadyToScan"),
            ScanningStatus::UnableToScan => String::from("UnableToScan"),
        }
    }
}

#[component]
pub(crate) fn QRReader() -> Element {
    let mut qr_value = use_signal(|| "no qr code scanned yet".to_string());

    let mut scanning_status = use_signal(|| ScanningStatus::UnableToScan.to_string());

    let mut protocol = use_signal(String::new);

    // Initialize NFC status on mount
    use_effect(move || {
        spawn(async move {
            match document::eval(
                r#"
                    if ("BarcodeDetector" in window) {
                        return true
                    } else {
                        return false
                    }
                "#
            ).await {
                Ok(result) => {
                    if let Some(val) = result.as_bool() {
                        if val == true {
                            scanning_status.set(ScanningStatus::ReadyToScan.to_string());
                        }
                    }
                }
                Err(e) => eprintln!("Error checking BarcodeDetector: {}", e),
            }
        });
    });

    // check for http or https
    use_effect(move || {
        spawn(async move {
            match document::eval(
                r#"return window.location.protocol"#
            ).await {
                Ok(result) => {
                    protocol.set(result.to_string());
                }
                Err(e) => eprintln!("Error checking Protocol: {}", e),
            }
        });
    });

    rsx!{
        button {
            id: "scan-btn",
            onclick: move |_| async move {
                scanning_status.set(ScanningStatus::Scanning.to_string());
                match document::eval(
                    r#"
                        const camStream = document.getElementById('cam-stream');
                        const scanBtn = document.getElementById('scan-btn');

                        const stopStream = (streamToStop) => {
                            streamToStop.getTracks().forEach((track) => track.stop());
                            camStream.style.display = "none";
                            camStream.srcObject = null;
                            //scanBtn.textContent = "Click me to scan QR Code";
                        };

                        try {
                            camStream.style.display = "block";

                            // Gets camera stream
                            stream = await navigator.mediaDevices.getUserMedia({
                                video: {
                                    facingMode: { ideal: "environment" }
                                },
                                audio: false
                            });
                            camStream.srcObject = stream;
                            await camStream.play();
                            //scanBtn.textContent = "Scanning...";

                            // Creates new barcode detection instance
                            const barcodeDetector = new BarcodeDetector({ formats: ["code_128", "code_39", "code_93", "qr_code"]});
                            let detected = false;

                            // Runs detect() every second till a barcode is captured
                            const intervalID = setInterval(async () => {
                                // Detects the contents of the camera stream to capture any visible barcode
                                const barcodes = await barcodeDetector.detect(camStream);
                                const lastBarcode = barcodes[barcodes.length -1];
                                // If barcode detected, stop stream and display result
                                // If this runs, setTimeout callback doesn't run
                                if (barcodes.length > 0) {
                                    stopStream(stream);
                                    // display only the last detected barcode
                                    // displayResult(lastBarcode.rawValue, lastBarcode.format);
                                    detected = true;
                                    clearInterval(intervalID);
                                    dioxus.send(lastBarcode.rawValue);
                                }
                            }, 1000);

                            // Stop stream if nothing has been detected after 15 seconds
                            setTimeout(() => {
                                if (!detected) {
                                    stopStream(stream);
                                    console.error("Barcode Not Detected or Format Not Supported");
                                    clearInterval(intervalID);
                                }}, 15000);

                        } catch (e) {
                            console.error(e.message);
                        }
                    "#
                )
                .recv::<String>()
                .await {
                    Ok(values) => {
                        qr_value.set(values);
                        scanning_status.set(ScanningStatus::ReadyToScan.to_string());
                    }
                    Err(error) => {
                        tracing::error!("Error reading from barcodeDetector: {}", error);
                    }
                }
            },
            disabled: {
                    if scanning_status() == ScanningStatus::ReadyToScan.to_string() || scanning_status() == ScanningStatus::Scanning.to_string() {
                        false
                    } else {
                        true
                    }
            },

            {
                if scanning_status() == ScanningStatus::ReadyToScan.to_string() {
                    "Click me to read QR Code"
                } else if scanning_status() == ScanningStatus::Scanning.to_string() {
                    "Reading QR Code"
                } else if scanning_status() == ScanningStatus::UnableToScan.to_string() {
                    "Unable to read QR Code"
                } else {
                    "Unknown error"
                }
            },
        }

        video {
            id: "cam-stream",
            width: "100%",
            height: "75%",
            display: "none",
        }

        Accordion { allow_multiple_open: false, horizontal: false,
            AccordionItem { index: 0,
                AccordionTrigger { "Debug Values" }
                AccordionContent {
                    div { padding_bottom: "1rem",

                        p { "Current protocol: {protocol} <-- For QR Code scanning https is required" }

                        if scanning_status() == ScanningStatus::ReadyToScan.to_string() || scanning_status() == ScanningStatus::Scanning.to_string() {
                            p { "BarcodeDetector available: Your phone or your browser probably does support qr code scanning" }
                        } else {
                            p { "BarcodeDetector not available: Your phone or your browser probably doesn't support qr code scanning" }
                        }

                        if scanning_status() == ScanningStatus::ReadyToScan.to_string() {
                            p { "Barcodedetector read: {qr_value}" }
                        }
                    }
                }
            }
        }
    }
}

I wrote this small component. Is limited to chromium browsers tho (tested on chrome android).

[brianmay]

Looks good. I should try it sometime. See if it is any better then my solution.

It should be possible to increase the browser support with use of an appropriate polyfill. e.g. https://github.com/undecaf/barcode-detector-polyfill. For some reason I found I needed this on all browsers, IIRC, including Chrome on Android on S24. Not sure why, that combination should have built in support I think. Might be misremembering.

I still had several issues.

1. Convincing barcode-detector-polyfill to use my copy of zbar.wasm. I do have a solution that works, it isn't great.
2. Barcodes are often miss-scanned. As in the number read is completely different of first attempt, then reads correctly on 2nd attempt. Not sure if this is my fault or a fault in the zbar.wasm. I note that an Android barcode scanner app I tried also had exactly the same issue.
3. After scanning a barcode, I couldn't work out how to get the webpage to release the camera. Chrome thinks the webpage is still using the camera, and makes it annoying to try to turn on the flash/light for example.

[Ekcum]

2. i only tested on small qr codes, maybe there are issues with barcodes, which im not aware of. (I will test it later)

3. i think your issue is that you dont stop the stream (only a guess, i just didn't have your issue) take the object which navigator.mediaDevices.getUserMedia returns and call stopStream() on it.

[brianmay]

3. Still testing, but I think most likely you are correct. Have added in appropriate place:

   video.srcObject.getTracks().forEach(track => track.stop());
   

   Seems like it might be working. For some reason I thought/hoped that removing the
   video element would be sufficient to stop streaming. But I think this is not the case.

I missed one issue:

4. Sometimes, auto-focus doesn't work, and can't focus on barcode. Need to switch to camera app, focus, then switch back and it works.

   As I typed this in, it occurred to me this could be connected somehow with my failure to stop the streaming. So will have to see if this still happens now.

[Ekcum]

4. as far i know you can adjust manually the focus. https://www.dynamsoft.com/codepool/camera-focus-control-on-web.html will add it later to my code and lets see how good it works. i think manually adjusting the focusdistance is actually a very good solution (autofocus is often broken)