[FEATURE REQUEST] ImageSource should allow WebGLTexture as input

[marcusx2]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

MacOS 13.6.3

MediaPipe Tasks SDK version

0.10.9

Task name (e.g. Image classification, Gesture recognition etc.)

Image classification

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

ImageSource does not support WebGLTexture as input

Describe the expected behaviour

ImageSource should support WebGLTexture as input

Standalone code/steps you may have used to try to get what you need

ImageSource

does not support WebGLTexture as input. It looks like I have to convert it first using something like this https://stackoverflow.com/a/18804083/8509272 to be able to use mediapipe with webgl textures. Or am I missing something?

In addition, the ObjectDetectorOptions interface: https://developers.google.com/mediapipe/api/solutions/js/tasks-vision.objectdetectoroptions

is said to extend VisionTaskOptions and ClassifierOptions, but both are undocumented.

Other info / Complete Logs

No response

[marcusx2]

Is there an alternative way to deal WebGLTexture as input that doesn't require conversion?

[schmidt-sebastian]

Your best bet is to convert the WebGLTexture to an ImageBitmap for now.

[marcusx2]

Any particular reason for creating an imageBitmap if I can use an imageData? Is it more efficient? Seems like to use an imageBitmap I have to use an imageData first, so it's an extra step.

Also would it be possible to use DrawingUtils with the imageBitmap or imageData?

[schmidt-sebastian]

You should be able to transfer GPU-to-GPU with an ImageBitmap: https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas/transferToImageBitmap. This is much cheaper than creating a CPU based ImageData.

Yes, DrawingUtils supports GPU and CPU based masks.

[marcusx2]

Last question, I have something like this

GLctx.bindTexture(GLctx.TEXTURE_2D, GL.textures[texturePtr]);
GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.RGBA, GLctx.RGBA, GLctx.UNSIGNED_BYTE, source);

How can I use the transferToImageBitmap at this point? I already have the gl context, I can't create a new one with offscreen.getContext("webgl")(also it looks like getContext("webgl") isn't supported on Safari?) Thanks a lot!

[schmidt-sebastian]

Once you draw the texture to a canvas you should be able to use the transferToImageBitmap() function.