[Mobile] [react-native] [android] Tensor.fromImage error ReferenceError: Property 'document' doesn't exist

[hungtooc]

Describe the issue

I tried to convert image to tensor, but none of method works
node_modules/onnxruntime-common/lib/tensor-factory.ts

export interface TensorFactory {
  /**
   * create a tensor from an ImageData object
   *
   * @param imageData - the ImageData object to create tensor from
   * @param options - An optional object representing options for creating tensor from ImageData.
   *
   * The following default settings will be applied:
   * - `tensorFormat`: `'RGB'`
   * - `tensorLayout`: `'NCHW'`
   * - `dataType`: `'float32'`
   * @returns A promise that resolves to a tensor object
   */
  fromImage(imageData: ImageData, options?: TensorFromImageDataOptions):
      Promise<TypedTensor<'float32'>|TypedTensor<'uint8'>>;

  /**
   * create a tensor from a HTMLImageElement object
   *
   * @param imageElement - the HTMLImageElement object to create tensor from
   * @param options - An optional object representing options for creating tensor from HTMLImageElement.
   *
   * The following default settings will be applied:
   * - `tensorFormat`: `'RGB'`
   * - `tensorLayout`: `'NCHW'`
   * - `dataType`: `'float32'`
   * @returns A promise that resolves to a tensor object
   */
  fromImage(imageElement: HTMLImageElement, options?: TensorFromImageElementOptions):
      Promise<TypedTensor<'float32'>|TypedTensor<'uint8'>>;

  /**
   * create a tensor from URL
   *
   * @param urlSource - a string as a URL to the image or a data URL containing the image data.
   * @param options - An optional object representing options for creating tensor from URL.
   *
   * The following default settings will be applied:
   * - `tensorFormat`: `'RGB'`
   * - `tensorLayout`: `'NCHW'`
   * - `dataType`: `'float32'`
   * @returns A promise that resolves to a tensor object
   */
  fromImage(urlSource: string, options?: TensorFromUrlOptions): Promise<TypedTensor<'float32'>|TypedTensor<'uint8'>>;

  /**
   * create a tensor from an ImageBitmap object
   *
   * @param bitMap - the ImageBitmap object to create tensor from
   * @param options - An optional object representing options for creating tensor from URL.
   *
   * The following default settings will be applied:
   * - `tensorFormat`: `'RGB'`
   * - `tensorLayout`: `'NCHW'`
   * - `dataType`: `'float32'`
   * @returns A promise that resolves to a tensor object
   */
  fromImage(bitmap: ImageBitmap, options: TensorFromImageBitmapOptions):
      Promise<TypedTensor<'float32'>|TypedTensor<'uint8'>>;
}

Error logs:

ReferenceError: Property 'document' doesn't exist
ReferenceError: Property 'document' doesn't exist
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:125611:24)
    at tryCallTwo (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:82:14)
    at ?anon_0_ (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:125610:27)
    at next (native)
    at asyncGeneratorStep (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28354:26)
    at _next (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28373:29)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28378:14)
    at tryCallTwo (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28370:25)
    at apply (native)
    at tensorFromImage (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:125639:24)
    at ?anon_0_ (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:125181:97)
    at next (native)
    at asyncGeneratorStep (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28354:26)
    at _next (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28373:29)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28378:14)
    at tryCallTwo (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28370:25)
    at apply (native)
    at fromImage (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:125184:34)
    at ?anon_0_ (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:123192:110)
    at next (native)
    at asyncGeneratorStep (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28354:26)
    at _next (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28373:29)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28378:14)
    at tryCallTwo (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/packages/react-native/ReactAndroid/hermes-engine/.cxx/Release/554c6x4t/x86_64/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:28370:25)
    at apply (native)
    at loadImage (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:123200:29)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:123203:16)
    at commitHookEffectListMount (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:95199:38)
    at commitPassiveMountOnFiber (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:96400:44)
    at commitPassiveMountEffects_complete (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:96372:40)
    at commitPassiveMountEffects_begin (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:96362:47)
    at commitPassiveMountEffects (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:96352:40)
    at flushPassiveEffectsImpl (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:98028:34)
    at flushPassiveEffects (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:97983:43)
    at performSyncWorkOnRoot (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:97273:28)
    at flushSyncCallbacks (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:86472:36)
    at flushSyncCallbacksOnlyInLegacyMode (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:86454:29)
    at scheduleUpdateOnFiber (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:96841:47)
    at updateContainer (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:99488:32)
    at render (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:100044:24)
    at renderElement (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:4399:77)
    at renderApplication (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:59322:27)
    at run (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:58704:42)
    at runApplication (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:58760:28)
    at apply (native)
    at __callFunction (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:3318:36)
    at anonymous (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:3038:31)
    at __guard (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:3255:15)
    at callFunctionReturnFlushedQueue (http://10.0.2.2:8081/index.bundle//&platform=android&dev=true&minify=false&app=com.sdc.face&modulesOnly=false&runModule=true:3037:21)

To reproduce

const imageTensor = await Tensor.fromImage('file://' + RNFS.DocumentDirectoryPath + '/image.jpg',);

Urgency

yes

Platform

Android

OS Version

android

ONNX Runtime Installation

Released Package

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

onnxruntime-react-native

ONNX Runtime Version or Commit ID

latest

ONNX Runtime API

JavaScript

Architecture

Other / Unknown

Execution Provider

Default CPU

Execution Provider Library Version

No response

[skottmckay]

Guessing that is intended for usage in onnxruntime-web as the document instance would be the HTML DOM document.

@shalvamist @fs-eire are these helpers expected to work with web only?

What model are you trying to run? For usage in react-native one option would be to add pre-processing to the model so the conversion from image to model input is part of the model, as there are many steps besides converting to RGB (typically resizing with antialiasing, centered crop or letterboxing, normalization to convert to float values in a range of either 0..1 or -1..1 depending on where the model came from, transpose from channels last to channels first) that need to take place.

If that can be added to the model you can simply create a tensor from the jpg bytes and use that as model input.

https://github.com/microsoft/onnxruntime-extensions/blob/main/onnxruntime_extensions/tools/Example%20usage%20of%20the%20PrePostProcessor.md

Example usage for YOLO where the pre-processing does the image conversion etc. and the post-processing selects the best bounding boxes, draws them on the original image and returns a jpg.

https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/yolo_e2e.py

Side note: is using a file path directly on Android going to work either way? This comment in an unrelated issue suggests you need to do some extra work first.

[hungtooc]

@skottmckay
the error I'm getting is the input tensor preparation stage and I don't think it's related to the onnx I'm using.
As for model information, I use custom model retinaface, it's backbone is MobileNet-0.5 , you can download it here: https://drive.google.com/file/d/1M5685m-bKnMCt0u2myJoEK5gUY3TDt_1/view
I am using the onnxruntime-react-native to load and process images. Currently I loaded model:

import RNFS from 'react-native-fs';    
//
//
    const loadModel = async () => {
    // const temp = require('./mnist.ort');
    // console.log(temp);
    let path = RNFS.DocumentDirectoryPath + '/2d106det.onnx';
    await RNFS.downloadFile({
      fromUrl:
        'https://firebasestorage.googleapis.com/v0/b/cavina-ai-8e2e4.appspot.com/o/cavina%2Fmodels%2F2d106det.onnx?alt=media&token=2bfc2728-6c6d-4bda-851b-c1c46b45a0a4&_gl=1*16eolju*_ga*MTMzMTkwMDQ4MS4xNjgyOTM3MDA2*_ga_CW55HF8NVT*MTY5NjE1OTE5OS40MjcuMS4xNjk2MTU5NDA3LjU5LjAuMA..',
      toFile: path,
    });

    const session01 = await InferenceSession.create(
      'file://' + RNFS.DocumentDirectoryPath + '/2d106det.onnx',
    );
    console.log('model in/out:', session01.inputNames, session01.outputNames);
    return session01;
  };
  useEffect( () => {
    const model = loadModel();
  }, []);

[hungtooc]

and there is another problem, which is that the documentation on how to load the model from the path is not clear, and the example on how to load the mnist.ort model is quite complicated.

[fs-eire]

React-native uses JSC as JavaScript engine and there is no DOM available in this environment. Tensor.fromImage need a react-native specific implementation.

[hungtooc]

Hi @fs-eire, I'm not sure it is a #bug or #feature-request, hope it can support soon. thanks

[skottmckay]

Adding support for using Tensor.fromImage from React Native would be a new feature.

You can however add the image pre-processing to your model and all you need to do on the React Native side is read the bytes from the image and create a Tensor with those.

If it's a mobilenet based model you can use onnxruntime_extensions to add the pre-processing to the model.

https://github.com/microsoft/onnxruntime-extensions/blob/46a37c3902ee29c50906c89a558f47e15ea03b33/test/test_tools_add_pre_post_processing_to_model.py#L56

This is example code for reading bytes from an image into a tensor to use as model input once the pre-processing is part of the model:

https://github.com/microsoft/onnxruntime-inference-examples/blob/2f91d22e50b89545d9f2a39c7872d7ff0a77af96/mobile/examples/ort_extensions_react_native_usage/ort-ext-rn-usage/App.tsx#L11C7-L48

[kgeorge-fission]

Adding support for using Tensor.fromImage from React Native would be a new feature.

You can however add the image pre-processing to your model and all you need to do on the React Native side is read the bytes from the image and create a Tensor with those.

If it's a mobilenet based model you can use onnxruntime_extensions to add the pre-processing to the model.

https://github.com/microsoft/onnxruntime-extensions/blob/46a37c3902ee29c50906c89a558f47e15ea03b33/test/test_tools_add_pre_post_processing_to_model.py#L56

This is example code for reading bytes from an image into a tensor to use as model input once the pre-processing is part of the model:

https://github.com/microsoft/onnxruntime-inference-examples/blob/2f91d22e50b89545d9f2a39c7872d7ff0a77af96/mobile/examples/ort_extensions_react_native_usage/ort-ext-rn-usage/App.tsx#L11C7-L48

Hi @skottmckay , I'm trying to use this pre_post_processing to add a head for sam encoder so that it can take base64 image for a react native runtime. But this PrePostProcessor requires atleast opset 16 and react-native runtime supports only up to opset 15.

[skottmckay]

AFAIK the ORT react-native package includes all the operators and opsets. Version 1.17 was recently released.

Are you seeing an error using that that says only opset 15 is supported?

[kgeorge-fission]

I'm yet to test it in a react native runtime.
But when I use the check_onnx_model_mobile_usability tool from onnxruntime, it says :

Model uses ONNX opset 16.
The pre-built package only supports ONNX opsets [12, 13, 14, 15].

This is from a dummy model I tried to add the pre_processing.

But when I add a bytes input head to sam encoder model using the pre_processing, I get:

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Node (/Transpose) Op (Transpose) [TypeInferenceError] Invalid attribute perm {2, 0, 1}, input shape = {3}

[skottmckay]

The ORT react native package has included a full build of ORT since v1.13, so that message isn't applicable any more.

Sounds like something is off in the pre-processing. You have a Transpose node that is trying to re-order 3D input (e.g. HWC to CHW), but the input is a 1D tensor with 3 elements (guessing that's the shape of the data and not the actual data).

[kgeorge-fission]

Thanks @skottmckay , glad to know opset versions are not an issue anymore.
I'm not sure what the error is, I'll check if there are any issues with the way I'm adding the preprocessing head.

[shaquibimdad]

Is there any progress i am trying to create Tensor from image and is throwing Error: Canvas is not supported as canvas is not supported in react-native i want to inference u2net model

Model Inputs:
Name: input.1, Shape: [1, 3, 320, 320], Type: float32

Model Outputs:
Name: 1959, Shape: [1, 1, 320, 320], Type: float32
Name: 1960, Shape: [1, 1, 320, 320], Type: float32

model is loaded and inferencing with test tensors but i am not able to find proper way to convert image to tensor and vice versa in react native