Understanding the Screen Share Protocol

The screen share protocol in Ulterius works by looking for differences in individual frames of the desktop (think P-frames). These changes are then sent to the client as patches which are then stitched into the canvas.

Screen capture data is packed into the following binary format:

-----Body-----
Guid (byte[16])
Bounds X (int32)
Bounds Y (int32)
Bounds Top (int32)
Bounds Bottom (int32)
Bounds Left (int32)
Bounds Right (int32)
Jpeg Patch (byte[<remaining>])
----------

For example, the web client uses this parse function (using the jDataView library):

export function unpackFrameData(data: Uint8Array) {
    let fv = new jDataView(data, 0, data.length, true)
    let uid = fv.getString(16) // unused
    let x = fv.getInt32()
    let y = fv.getInt32()
    let top = fv.getInt32()
    let bottom = fv.getInt32()
    let left = fv.getInt32()
    let right = fv.getInt32()
    let image = decompressData(data.subarray(16 + (4*6)))
    return {
        x, y, top, bottom, left, right, image
    }
}

When starting an Ulterius screen share stream these patches are sent to the client in real time.

You can also request a patch consisting of the entire screen (I-frame) by using the fullframe endpoint. The full frame response comes in a JSON format instead of binary:

declare interface FrameData {
    screenBounds: {
        top: number,
        bottom: number,
        left: number,
        right: number,
        x: number,
        y: number,
        height: number,
        isEmpty: boolean,
    },
    frameData: number[] //byte array; the actual image
}