Add developer documentation

And update README.

Author: rom1v

DEVELOP.md

@@ -0,0 +1,238 @@
+# scrcpy for developers
+
+## Overview
+
+This application is composed of two parts:
+ - the server (`scrcpy-server.jar`), to be executed on the device,
+ - the client (the `scrcpy` binary), executed on the host computer.
+
+The client is responsible to push the server to the device and start its
+execution.
+
+Once the client and the server are connected to each other, the server initially
+sends device information (name and initial screen dimensions), then starts to
+send a raw H.264 video stream of the device screen. The client decodes the video
+frames, and display them as soon as possible, without buffering, to minimize
+latency. The client is not aware of the device rotation (which is handled by the
+server), it just knows the dimensions of the video frames.
+
+The client captures relevant keyboard and mouse events, that it transmits to the
+server, which injects them to the device.
+
+
+
+## Server
+
+
+### Privileges
+
+Capturing the screen requires some privileges, which are granted to `shell`.
+
+The server is a Java application (with a [`public static void main(String...
+args)`][main] method), compiled against the Android framework, and executed as
+`shell` on the Android device.
+
+[main]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/Server.java#L61
+
+To run such a Java application, the classes must be [_dexed_][dex] (typically,
+to `classes.dex`). If `my.package.MainClass` is the main class, compiled to
+`classes.dex`, pushed to the device in `/data/local/tmp`, then it can be run
+with:
+
+    adb shell CLASSPATH=/data/local/tmp/classes.dex \
+        app_process / my.package.MainClass
+
+_The path `/data/local/tmp` is a good candidate to push the server, since it's
+readable and writable by `shell`, but not world-writable, so a malicious
+application may not replace the server just before the client executes it._
+
+Instead of a raw _dex_ file, `app_process` accepts a _jar_ containing
+`classes.dex` (e.g. an [APK]). For simplicity, and to benefit from the gradle
+build system, the server is built to an (unsigned) APK (renamed to
+`scrcpy-server.jar`).
+
+[dex]: https://en.wikipedia.org/wiki/Dalvik_(software)
+[apk]: https://en.wikipedia.org/wiki/Android_application_package
+
+
+### Hidden methods
+
+Although compiled against the Android framework, [hidden] methods and classes are
+not directly accessible (and they may differ from one Android version to
+another).
+
+They can be called using reflection though. The communication with hidden
+components is provided by [_wrappers_ classes][wrappers] and [aidl].
+
+[hidden]: https://stackoverflow.com/a/31908373/1987178
+[wrappers]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/wrappers
+[aidl]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/aidl/android/view
+
+
+### Threading
+
+The server uses 2 threads:
+
+ - the **main** thread, encoding and streaming the video to the client;
+ - the **controller** thread, listening for _control events_ (typically,
+   keyboard and mouse events) from the client.
+
+Since the video encoding is typically hardware, there would be no benefit in
+encoding and streaming in two different threads.
+
+
+### Screen video encoding
+
+The encoding is managed by [`ScreenEncoder`].
+
+The video is encoded using the [`MediaCodec`] API. The codec takes its input
+from a [surface] associated to the display, and writes the resulting H.264
+stream to the provided output stream (the socket connected to the client).
+
+[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java
+[`MediaCodec`]: https://developer.android.com/reference/android/media/MediaCodec.html
+[surface]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L63-L64
+
+On device [rotation], the codec, surface and display are reinitialized, and a
+new video stream is produced.
+
+New frames are produced only when changes occur on the surface. This is good
+because it avoids to send unnecessary frames, but there are drawbacks:
+
+ - it does not send any frame on start if the device screen does not change,
+ - after fast motion changes, the last frame may have poor quality.
+
+Both problems are [solved][repeat] by the flag
+[`KEY_REPEAT_PREVIOUS_FRAME_AFTER`][repeat-flag].
+
+[rotation]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L89-L92
+[repeat]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L125-L126
+[repeat-flag]: https://developer.android.com/reference/android/media/MediaFormat.html#KEY_REPEAT_PREVIOUS_FRAME_AFTER
+
+
+### Input events injection
+
+_Control events_ are received from the client by the [`EventController`] (run in
+a separate thread). There are 5 types of input events:
+ - keycode (cf [`KeyEvent`]),
+ - text (special characters may not be handled by keycodes directly),
+ - mouse motion/click,
+ - mouse scroll,
+ - custom command (e.g. to switch the screen on).
+
+All of them may need to inject input events to the system. To do so, they use
+the _hidden_ method [`InputManager.injectInputEvent`] (exposed by our
+[`InputManager` wrapper][inject-wrapper]).
+
+[`EventController`]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/EventController.java#L70
+[`KeyEvent`]: https://developer.android.com/reference/android/view/KeyEvent.html
+[`MotionEvent`]: https://developer.android.com/reference/android/view/MotionEvent.html
+[`InputManager.injectInputEvent`]: https://android.googlesource.com/platform/frameworks/base/+/oreo-release/core/java/android/hardware/input/InputManager.java#857
+[inject-wrapper]: https://github.com/Genymobile/scrcpy/blob/v1.0/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L27
+
+
+
+## Client
+
+The client relies on [SDL], which provides cross-platform API for UI, input
+events, threading, etc.
+
+The video stream is decoded by [libav] (FFmpeg).
+
+[SDL]: https://www.libsdl.org
+[libav]: https://www.libav.org/
+
+### Initialization
+
+On startup, in addition to _libav_ and _SDL_ initialization, the client must
+push and start the server on the device, and open a socket so that they may
+communicate.
+
+Note that the client-server roles are expressed at the application level:
+
+ - the server _serves_ video stream and handle requests from the client,
+ - the client _controls_ the device through the server.
+
+However, the roles are inverted at the network level:
+
+ - the client opens a server socket and listen on a port before starting the
+   server,
+ - the server connects to the client.
+
+This role inversion guarantees that the connection will not fail due to race
+conditions, and avoids polling.
+
+Once the server is connected, it sends the device information (name and initial
+screen dimensions). Thus, the client may init the window and renderer, before
+the first frame is available.
+
+To minimize startup time, SDL initialization is performed while listening for
+the connection from the server (see commit [90a46b4]).
+
+[90a46b4]: https://github.com/Genymobile/scrcpy/commit/90a46b4c45637d083e877020d85ade52a9a5fa8e
+
+
+### Threading
+
+The client uses 3 threads:
+
+ - the **main** thread, executing the SDL event loop,
+ - the **decoder** thread, decoding video frames,
+ - the **controller** thread, sending _control events_ to the server.
+
+
+### Decoder
+
+The [decoder] runs in a separate thread. It uses _libav_ to decode the H.264
+stream from the socket, and notifies the main thread when a new frame is
+available.
+
+There are two [frames] simultaneously in memory:
+ - the **decoding** frame, written by the decoder from the decoder thread,
+ - the **rendering** frame, rendered in a texture from the main thread.
+
+When a new decoded frame is available, the decoder _swaps_ the decoding and
+rendering frame (with proper synchronization). Thus, it immediatly starts
+to decode a new frame while the main thread renders the last one.
+
+[decoder]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/decoder.c
+[frames]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/frames.h
+
+
+### Controller
+
+The [controller] is responsible to send _control events_ to the device. It runs
+in a separate thread, to avoid I/O on the main thread.
+
+On SDL event, received on the main thread, the [input manager][inputmanager]
+creates appropriate [_control events_][controlevent]. It is responsible to
+convert SDL events to Android events (using [convert]). It pushes the _control
+events_ to a blocking queue hold by the controller. On its own thread, the
+controller takes events from the queue, that it serializes and sends to the
+client.
+
+[controller]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/controller.h
+[controlevent]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/controlevent.h
+[inputmanager]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/inputmanager.h
+[convert]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/convert.h
+
+
+### UI and event loop
+
+Initialization, input events and rendering are all [managed][scrcpy] in the main
+thread.
+
+Events are handled in the [event loop], which either updates the [screen] or
+delegates to the [input manager][inputmanager].
+
+[scrcpy]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/scrcpy.c
+[event loop]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/scrcpy.c#L38
+[screen]: https://github.com/Genymobile/scrcpy/blob/v1.0/app/src/screen.h
+
+
+## Hack
+
+For more details, go read the code!
+
+If you find a bug, or have an awesome idea to implement, please discuss and
+contribute ;-)

README.md

@@ -1,8 +1,8 @@
 # scrcpy
 
-This project displays screens of Android devices connected on USB, and allow to
-control them using the host mouse and keyboard. It does not require any _root_
-access. It works on _GNU/Linux_, _Windows_ and _Mac OS_.
+This application provides display and control of Android devices connected on
+USB. It does not require any _root_ access. It works on _GNU/Linux_, _Windows_
+and _Mac OS_.
 
 ![screenshot](assets/screenshot-debian-600.jpg)
 
@@ -50,8 +50,15 @@ Install the required packages from your package manager (here, for Debian):
 
 #### Windows
 
-For Windows, for simplicity, a prebuilt package with all the dependencies
-(including `adb`) is available: TODO.
+For Windows, for simplicity, a prebuilt archive with all the dependencies
+(including `adb`) is available:
+
+ - [`scrcpy-windows-with-deps-v1.0.zip`][direct-windows-with-deps].  
+   _(SHA-256: TODO)_
+
+[direct-windows-with-deps]: https://github.com/Genymobile/scrcpy/releases/download/v1.0/scrcpy-windows-with-deps-v1.0.zip
+
+_(It's just a portable version including _dll_ copied from MSYS2.)_
 
 Instead, you may want to build it manually. You need [MSYS2] to build the
 project. From an MSYS2 terminal, install the required packages:
@@ -128,7 +135,12 @@ Just remove them to "uninstall" the application.
 
 Since the server binary, that will be pushed to the Android device, does not
 depend on your system and architecture, you may want to use the prebuilt binary
-instead: [`scrcpy-server.jar`](TODO).
+instead:
+
+ - [`scrcpy-server-v1.0.jar`][direct-scrcpy-server].  
+   _(SHA-256: TODO)_
+
+[direct-scrcpy-server]: https://github.com/Genymobile/scrcpy/releases/download/v1.0/scrcpy-server-v1.0.jar
 
 In that case, the build does not require Java or the Android SDK.
 
@@ -186,6 +198,23 @@ If several devices are listed in `adb devices`, you must specify the _serial_:
  | enable/disable FPS counter (on stdout) | `Ctrl`+`i`    |
 
 
+## Why _scrcpy_?
+
+A colleague challenged me to find a name as unpronounceable as [gnirehtet].
+
+[`strcpy`] copies a **str**ing; `scrcpy` copies a **scr**een.
+
+[gnirehtet]: https://github.com/Genymobile/gnirehtet
+[`strcpy`]: http://man7.org/linux/man-pages/man3/strcpy.3.html
+
+
+## Developers
+
+Read the [developers page].
+
+[developers page]: DEVELOP.md
+
+
 ## Licence
 
     Copyright (C) 2018 Genymobile