-ssl option fails with rfb error

[iouwon]

When i start x11vnc with -ssl and an incoming connection from xrdp occurs it fails with 'invalid version rfb 003.003'

The x11vnc log is below but I have no idea what rfb is or how to get a valid version, but apparently there is a version 3.8:
02/08/2019 01:55:34 SSL: accept_openssl(OPENSSL_VNC6)
02/08/2019 01:55:34 check_access: client addr ::1 is local.
02/08/2019 01:55:34 SSL: spawning helper process to handle: ::1:55814
02/08/2019 01:55:34 SSL: helper for peerport 55814 is pid 11580:
02/08/2019 01:55:34 connect_tcp: trying: 127.0.0.1 20000
02/08/2019 01:55:36 check_vnc_tls_mode: waited: 1.422010 / 1.40 input: (future) RFB Handshake
02/08/2019 01:55:36 check_vnc_tls_mode: version: 3.3
02/08/2019 01:55:36 check_vnc_tls_mode: invalid version: 'RFB 003.003
'
02/08/2019 01:55:36 SSL: ssl_helper[11580]: exit case 2 (ssl_init failed)
02/08/2019 01:55:36 SSL: accept_openssl: cookie from ssl_helper[11580] FAILED. 0

Ubuntu 18.04:

• Xorg version used: 1.20.4

[iouwon]

FYI, i tried the -rfbversion 3.8 and got the same error

[iouwon]

I read that xrdp uses 3.3 and it seems there's no way to configure that so I tried -rfbversion 3.3 to force x11vnc to use 3.3... and same error

[bk138]

Have you tried with different x11vnc versions and/or different VNC clients?

[iouwon]

If i use the disable ssl env param and exclude the -ssl cmd line option when i run x11vnc it connects just fine and i can control the session remotely. I have also set up an loopback ssl tunnel to connect xrdp to x11vnc and this also works but i'd like to be able to create a secure connection between xrdp and x11vnc ad hoc

I've not tried any other version and the version I'm currently on is:
x11vnc: 0.9.13 lastmod: 2011-08-10

[EzraBrooks]

I'm experiencing this same problem with all clients.

[nerdfever]

Same here, using either tightvnc or VNConnect.

It appears x11vnc has an incompatible set of encryption options with...anything else.