Android support based on scrcpy

[RaiMan]

My suggestion is to start basic implementations against scrcpy in the class:
org.sikuli.script.support.Devices.Andoid.Device
which implements IScreen and IRobot - so everything for now is in one place.

Feel free, to create necessary supporting classes in the package org.sikuli.script.support.Devices.Andoid

Any comments, suggestions, ... please in this discussion issue.

[TeofilisMartisius]

Hi,

Sorry for a long delay, I was unable to work on my hobby projects.

I made some small progress. I managed to turn scrcpy into a shared library (modified meson build). Then I managed to produce a Java/JNI wrapper for scrcpy library using Bytedeco JavaCPP and invoke scrcpy from Java as a shared library.

So far nothing much works and the code is a mess, but it demonstrates the concept should work. Next thing to do is to export more functions from scrcpy library and to try to get some screen grabs from Java. I'll continue to work on this as I have time.

UPDATE. I can see the screen running form Java via JavaCPP via libscrcpy, everything starts fine. Now I need to tweak scrcpy to decode h264 video stream into my Java bitmap instead of displaying it, and I need to tweak scrcpy to take events from Java code instead of mouse on window. Then clean up the code. Good progress so far.

Teofilis

[RaiMan]

No hurry ;-)
Anyways this would be a very welcome goody for some people out there.

[TeofilisMartisius]

Ok, progress update.

I have refactored scrcpy enough to be able to hook into video feed. Scrcpy is using ffmpeg library under the hood, and ffmpeg is available on Java as well with Bytedeco mappings. Long story short, now I have a Java process that is able to invoke scrcpy and get the video feed IN JAVA, as a BufferedImage.

I guess it's time to clean up the code and to get in touch with scrcpy guys and ask them if they would consider taking my changes upstream and making scrcpy into a library. I don't want to maintain this as a fork of scrcpy forever.

Teofilis

[RaiMan]

Ok, now I finally understand your concept.

Wish you all the best, to get the people at scrcpy convinced.

[TeofilisMartisius]

Ok, just FYI, I committed my code which uses scrcpy from Java, just to give you an idea of what's possible: https://github.com/TeofilisMartisius/scrcpy-java

I'm still waiting for more detailed feedback from scrcpy team with regards to merging my PR or whatever effort is needed to turn scrcpy into a library.

Next I'll do event passing (keys, mouse) Java => scrcpy => Android. Then some performance checks. And then try to integrate it all with SikuliX. Hopefully at some point the PR will get merged with upstream scrcpy, if not I can maintain the fork for now.

[RaiMan]

Thanks. Will have a look.

Just to track your activities on scrcpy: Genymobile/scrcpy#2595

[TeofilisMartisius]

Update. I did some more work cleaning up https://github.com/TeofilisMartisius/scrcpy-java Screen capture and Mouse clicks now work (drag still doesn't, and neither does keyboard).

Next thing I'll start wrapping SikuliX IScreen around a Scrcpy session and see how that goes.

Also, I'm using native libraries and for now the only platform supported is x86_64 Linux. If this takes off, we need to support more platforms...

Also, no more feedback on upstreaming this to scrcpy.

[RaiMan]

Thanks for the update.