Faucet v 1.10.4 "SSL WRONG VERSION NUMBER"

[lhchang]

Hello. We are trying to run Faucet in our Kubernetes cluster (v 1.10.4) and running into issues when trying to load the certs/private keys. We are running Faucet with the following args:

--ryu-ctl-privkey /etc/ryu/ssl/ControllerCertPrivateKey.pem --ryu-ctl-cert /etc/ryu/ssl/ControllerCert.pem --ryu-ca-certs /etc/ryu/ssl/RootCAcert.pem

These were different versus the ones in this example (https://docs.faucet.nz/en/1.10.4/installation.html?highlight=ssl#additional-arguments) but that was what was available in the listed available args. After running, we encounter this error.

loading app faucet.faucet
loading app os_ken.controller.ofp_handler
instantiating app None of DPSet
creating context dpset
instantiating app faucet.faucet of Faucet
instantiating app os_ken.controller.ofp_handler of OFPHandler
May 10 21:40:27 faucet INFO     version 1.10.4
May 10 21:40:27 faucet INFO     Reloading configuration
May 10 21:40:27 faucet INFO     configuration /etc/faucet/faucet.yaml changed, analyzing differences
May 10 21:40:27 faucet INFO     Add new datapath DPID 208962001918 (0x30a71b1bfe)
(1) wsgi starting up on http://0.0.0.0:9302
hub: uncaught exception: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/os_ken/lib/hub.py", line 69, in _launch
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/os_ken/lib/hub.py", line 150, in wrap_and_handle_ctx
    handle(ctx.wrap_socket(sock, **ssl_args), addr)
  File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 446, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 140, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 312, in do_handshake
    return self._call_trampolining(
  File "/usr/local/lib/python3.9/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
    return func(*a, **kw)
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)

Some things we have tried:

• Rolling back to a older version of Faucet
• We have checked that our ControllerCert.pem format is valid through openssl x509 -in ControllerCert.pem -text -noout
• Checking pub/priv keys match with openssl pkey -in ControllerCertPrivateKey.pem -pubout -outform pem | sha256sum and
  openssl x509 -in ControllerCert.pem -pubkey -noout -outform pem | sha256sum

Thank you

[anarkiwi]

Thanks for reporting this! Please would you try the self signed example at https://eventlet.net/doc/ssl.html? We'll also investigate.

[lhchang]

Thanks anarkiwi.

Client side:

nc 127.0.0.1 8443
asdf

Running the Python code with our key/certs:

Traceback (most recent call last):
  File "mytest.py", line 22, in <module>
    print(client_conn.read(100))
  File "/.local/lib/python3.8/site-packages/eventlet/green/OpenSSL/SSL.py", line 55, in read
    return self.fd.read(size)
  File "/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1865, in recv
    self._raise_ssl_error(self._ssl, result)
  File "/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1700, in _raise_ssl_error
    _raise_current_error()
  File "/.local/lib/python3.8/site-packages/OpenSSL/_util.py", line 55, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', '', 'wrong version number')]

[gizmoguy]

I think you're getting this error because you're trying to connect to an SSL/TLS socket with a plain-text connection.

Can you try using openssl s_client instead of nc to talk to your encrypted socket listening on port 8443:

openssl s_client -connect localhost:8443

[lhchang]

Ah you are right, thanks for the suggestion. Heres my output when using openssl s_client

openssl s_client -connect localhost:8443
CONNECTED(00000003)
140234030507328:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 283 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

On the server side:

Traceback (most recent call last):
  File "mytest.py", line 22, in <module>
    print(client_conn.read(100))
  File "/home/.local/lib/python3.8/site-packages/eventlet/green/OpenSSL/SSL.py", line 55, in read
    return self.fd.read(size)
  File "/home/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1865, in recv
    self._raise_ssl_error(self._ssl, result)
  File "/home/.local/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1700, in _raise_ssl_error
    _raise_current_error()
  File "/home/.local/lib/python3.8/site-packages/OpenSSL/_util.py", line 55, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', '', 'no shared cipher')]

Is this indicating the cert itself maybe missing something?

[gizmoguy]

Sounds like there is a mismatch in ciphers between your version of openssl and pyopenssl/eventlet.

I tried running the test code below with python 3.8.10 on ubuntu 20.04, using eventlet==0.33.1 and pyOpenSSL==22.0.0:

https://gist.github.com/gizmoguy/0372afad6e34170a693751859e3c0302

And it worked fine, could you confirm what version of linux you are using and your python / python library versions?:

$ python3 eventlet-ssl-test.py
b'HELLO\n'

$ openssl s_client -connect localhost:8443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=66:EE certificate key too weak
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
 0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
   i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICZjCCAc+gAwIBAgIUHBS3xl7XeAChb+hPzRHtvTL7y5owDQYJKoZIhvcNAQEF
BQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMjA1MjUwMjA5MDhaFw0yMzA1
MjUwMjA5MDhaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw
HwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwgZ8wDQYJKoZIhvcNAQEB
BQADgY0AMIGJAoGBALG8UnUQuDbWAOj5L3IZUYsSlbi3d+/zyCobw6l0EBDdSLZG
xdQrc80p6icAs16vV9VNkDEkvYh7iE/nDGreULvZp72qKltudEW2tKNIq43tvtCB
knTI6PohBr5s7re/jOEnKhZro8skNaRNacDRT1aAYElNRMNP/VhMhvCd2ctBAgMB
AAGjUzBRMB0GA1UdDgQWBBRe2ipeaKtXaLS6gV1iGPBZBZNUJTAfBgNVHSMEGDAW
gBRe2ipeaKtXaLS6gV1iGPBZBZNUJTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBBQUAA4GBAJsyLXMp4XUkSAy4xJAHwleNNWUzKAraJbxMKoLdmfZ3A7At16gj
XM1mPSX6A9dHpCdOMgJCgsfP7lwCEydtHaS/fh0mTGNJI9O8eEmJq4OBhXBhJqQj
rqf+5XEbcz6LSJlBkt4InLQ21HNCaSuLPvfDFHhQHiPXO9ayW6J0Johu
-----END CERTIFICATE-----
subject=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd

issuer=C = AU, ST = Some-State, O = Internet Widgits Pty Ltd

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1046 bytes and written 363 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 1024 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 744815FE250C33938A99B0C46435D093C91A2EC83987260D99680EFA27037218
    Session-ID-ctx: 
    Resumption PSK: 65B14DFC2B8E77A1C7DB6AC861CC4B75E994657EE8A48FDBDC210D38798D3290EE45EB83F742EA172A8532AF88667880
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 5f 87 a9 8e f0 65 b3 f3-3c dd 15 65 33 7c 67 98   _....e..<..e3|g.
    0010 - 15 f0 c2 ee 46 01 35 14-93 ae 9b f1 d1 dd bc 17   ....F.5.........
    0020 - fe b4 31 d1 0a 98 66 66-1d 2a 3d 4d f8 80 c9 b4   ..1...ff.*=M....
    0030 - 36 7d d5 ff 74 42 5c 16-bf 2b 70 09 94 b7 b1 d0   6}..tB\..+p.....
    0040 - cd 53 e7 28 11 39 4d 62-2f aa fd 4f 43 9e 40 59   .S.(.9Mb/..OC.@Y
    0050 - ab cb d4 6a 78 95 63 44-ff 2f 50 ca 5c bc 5d 5c   ...jx.cD./P.\.]\
    0060 - fa 25 cf b1 be b6 d5 e0-73 1a 2e 36 e0 f7 52 45   .%......s..6..RE
    0070 - d7 2e 27 13 af 64 6d 9a-f8 df 37 e7 b4 0b dc 8f   ..'..dm...7.....
    0080 - a9 70 e1 13 82 f9 6d f5-b9 b0 71 cc f2 09 73 77   .p....m...q...sw
    0090 - 18 1a e0 92 3d 78 db 66-8d 1b a4 6a a2 6c 42 1a   ....=x.f...j.lB.
    00a0 - 6a cd 13 ec 89 e0 7f 8f-8d b1 fa f9 0d 09 01 ba   j...............
    00b0 - af 2f 6d 29 44 b5 83 d9-66 ba 3f aa 21 8c a0 09   ./m)D...f.?.!...
    00c0 - 12 50 48 82 3d 67 4b b2-1f 3c bd da d4 3e 8a a4   .PH.=gK..<...>..

    Start Time: 1653444942
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 7A9161448040884056075587BF14A37B45B40855B24B141E376A876418E292FC
    Session-ID-ctx: 
    Resumption PSK: 056136C680B11569ECEB6398C57F45ACECC8E1AAB086675929065F20A0C12B83EB3AF3DCE950F886A7F711870E8221B9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 5f 87 a9 8e f0 65 b3 f3-3c dd 15 65 33 7c 67 98   _....e..<..e3|g.
    0010 - c0 30 c5 fd 84 af bf e8-9e 06 c8 8d a7 51 0d e2   .0...........Q..
    0020 - 85 b4 0b bd 77 7f 8b 1c-ec 4d 29 79 67 95 45 eb   ....w....M)yg.E.
    0030 - 6b fe 70 95 85 a1 b2 de-57 51 27 01 a6 0a aa 10   k.p.....WQ'.....
    0040 - 5c cb f7 64 00 cd 2a 65-31 12 b5 fa fe 85 1c c1   \..d..*e1.......
    0050 - 1e f2 ae a4 4f 18 78 55-86 bf bc 4e 2a 86 8a 3a   ....O.xU...N*..:
    0060 - b7 56 04 07 e2 fb 99 32-5e ed af 0f 36 44 a1 68   .V.....2^...6D.h
    0070 - 9a 71 d4 1c 1d 1d 6a a2-42 0d 3e 54 29 98 2d eb   .q....j.B.>T).-.
    0080 - 88 43 66 ff b7 fb 68 1b-c7 8a 5f 67 0b 91 14 7d   .Cf...h..._g...}
    0090 - b5 43 91 67 41 7c dd 35-98 d8 a7 cd 12 3c 29 f3   .C.gA|.5.....<).
    00a0 - cf 8a 71 ee 5d 18 ce 7d-33 da e8 11 41 f1 20 aa   ..q.]..}3...A. .
    00b0 - fa e0 b9 c9 5c 3c f1 32-5d 0d bf 6f 7e 43 d6 98   ....\<.2]..o~C..
    00c0 - 21 02 c0 fe 75 f7 f9 7f-6e 92 f9 ec e2 9c 1e d9   !...u...n.......

    Start Time: 1653444942
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
HELLO

[lhchang]

On our local machine using the eventlet example, it looks like we are running Python3.8, Ubuntu 20.04.1. The Faucet container seems to be using Python3.9, at least for 1.10.4.

[gizmoguy]

For your local machine running the eventlet example can you try upgrading to ubuntu 20.04.4? This is what I am using here and it works fine.

[lhchang]

I can confirm that using another generated cert that I can get the same results as your example output, but still not with the cert/keys we had on hand which were generated from another controller.

We tried pivoting a bit to see if we could get Faucet to talk TLS with OVS using the process described in https://docs.openvswitch.org/en/latest/howto/ssl/, we get the following output.

sudo faucet --ryu-ctl-privkey ctl-privkey.pem --ryu-ctl-cert ctl-cert.pem --ryu-ca-certs controllerca/cacert.pem --verbose
loading app faucet.faucet
loading app os_ken.controller.ofp_handler
instantiating app None of DPSet
creating context dpset
instantiating app faucet.faucet of Faucet
instantiating app os_ken.controller.ofp_handler of OFPHandler
BRICK dpset
  PROVIDES EventDP TO {'Faucet': {'dpset'}}
  PROVIDES EventDPReconnected TO {'Faucet': {'dpset'}}
  CONSUMES EventOFPStateChange
  CONSUMES EventOFPPortStatus
  CONSUMES EventOFPSwitchFeatures
BRICK Faucet
  CONSUMES EventFaucetEventSockHeartbeat
  CONSUMES EventFaucetMaintainStackRoot
  CONSUMES EventFaucetFastAdvertise
  CONSUMES EventFaucetAdvertise
  CONSUMES EventFaucetFastStateExpire
  CONSUMES EventFaucetStateExpire
  CONSUMES EventFaucetResolveGateways
  CONSUMES EventDP
  CONSUMES EventOFPDescStatsReply
  CONSUMES EventOFPErrorMsg
  CONSUMES EventOFPSwitchFeatures
  CONSUMES EventOFPFlowRemoved
  CONSUMES EventFaucetMetricUpdate
  CONSUMES EventOFPPacketIn
  CONSUMES EventOFPPortDescStatsReply
  CONSUMES EventOFPPortStatus
  CONSUMES EventDPReconnected
  CONSUMES EventReconfigure
BRICK ofp_event
  PROVIDES EventOFPStateChange TO {'dpset': {'dead', 'main'}}
  PROVIDES EventOFPPortStatus TO {'dpset': {'main'}, 'Faucet': {'main'}}
  PROVIDES EventOFPSwitchFeatures TO {'dpset': {'config'}, 'Faucet': {'config'}}
  PROVIDES EventOFPDescStatsReply TO {'Faucet': {'main'}}
  PROVIDES EventOFPErrorMsg TO {'Faucet': {'main'}}
  PROVIDES EventOFPFlowRemoved TO {'Faucet': {'main'}}
  PROVIDES EventOFPPacketIn TO {'Faucet': {'main'}}
  PROVIDES EventOFPPortDescStatsReply TO {'Faucet': {'config'}}
  CONSUMES EventOFPEchoReply
  CONSUMES EventOFPEchoRequest
  CONSUMES EventOFPErrorMsg
  CONSUMES EventOFPHello
  CONSUMES EventOFPPortDescStatsReply
  CONSUMES EventOFPPortStatus
  CONSUMES EventOFPSwitchFeatures
Terminated

Not sure if this is a related issue but it doesn't seem like the process stays up when fed the certs/keys from OVS PKI. Ideally we would like to get this to work with our original switch but if there is something I missed with the OVS method hopefully that may help with our original problem?

[gizmoguy]

That log doesn't have the error that occurred, what is in your faucet_exception.log file?

[lhchang]

Found the issue, had another instance running at the time. However after resolving that I am running back into another SSL issue:

hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/os_ken/lib/hub.py", line 69, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/os_ken/lib/hub.py", line 150, in wrap_and_handle_ctx
    handle(ctx.wrap_socket(sock, **ssl_args), addr)
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 445, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 139, in __init__
    self.do_handshake()
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 311, in do_handshake
    return self._call_trampolining(
  File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 161, in _call_trampolining
    return func(*a, **kw)
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1131)

This is using certs/keys generated from the OVS PKI tutorial

[gizmoguy]

What are you using to connect to the SSL/TLS port of faucet? netcat? openssl s_client? an openflow dataplane?

Have you checked whatever client you are using is correctly configured to speak SSL/TLS rather than plaintext?

[lhchang]

In the context of my previous comment we were attempting to use Open vSwitch to connect to Faucet. In the context of the original issue, we were trying to connect a physical OpenFlow switch to Faucet using certificates/keys pulled from another SDN controller that we had originally adopted the switches with. We had only used openssl s_client/netcat to test the Python eventlet code that was linked before.

[gizmoguy]

Can you try using openssl s_client to connect to the faucet SSL/TLS port instead of OVS?