Description
Describe the issue: When run against openssl 3, some of the tests that use the certificates in distributed/tests
(tls-cert.pem
et al) fail, with errors like "Client certificate validation failed: Basic Constraints of CA cert not marked critical". Per this page, this is due to constraints in newer versions of openssl that the current certificate files do not meet.
The last time the certificate generation script was really updated (apart from linting) appears to be 2021 (by openQA author @bmwiedemann ...small world!), and that was just to extend some validity periods. The last time it was really overhauled was f242303 in 2018. By comparison, the script it was based on - make_ssl_certs.py from cpython - has been updated several times to adapt to stricter constraints, including adding the basicConstraints = critical,CA:false
directive we need here.
Unfortunately the two scripts diverge enough that applying the updates to distributed's version isn't straightforward. I intend to look at it tomorrow, but am filing this in case someone else wants to get to it first, or I don't manage to do it.
Minimal Complete Verifiable Example:
Run the test suite with openssl 3.x, I think that should be enough. You'll get failures like:
________________ TestClientSecurityLoader.test_security_loader _________________
self = <distributed.comm.tcp.TLSConnector object at 0x7f671c814c20>
address = '0.0.0.0:38217', deserialize = True
connection_args = {'extra_conn_args': {}, 'require_encryption': True, 'ssl_context': <ssl.SSLContext object at 0x7f66af648f80>}
ip = '0.0.0.0', port = 38217
kwargs = {'ssl_options': <ssl.SSLContext object at 0x7f66af648f80>}
async def connect(self, address, deserialize=True, **connection_args):
self._check_encryption(address, connection_args)
ip, port = parse_host_port(address)
kwargs = self._get_connect_args(**connection_args)
try:
# server_hostname option (for SNI) only works with tornado.iostream.IOStream
if "server_hostname" in kwargs:
stream = await self.client.connect(
ip, port, max_buffer_size=MAX_BUFFER_SIZE
)
stream = await stream.start_tls(False, **kwargs)
else:
> stream = await self.client.connect(
ip, port, max_buffer_size=MAX_BUFFER_SIZE, **kwargs
)
../../BUILDROOT/usr/lib/python3.13/site-packages/distributed/comm/tcp.py:546:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.13/site-packages/tornado/tcpclient.py:292: in connect
stream = await stream.start_tls(
/usr/lib64/python3.13/site-packages/tornado/iostream.py:1367: in _do_ssl_handshake
self.socket.do_handshake()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <ssl.SSLSocket [closed] fd=-1, family=2, type=1, proto=0>, block = False
@_sslcopydoc
def do_handshake(self, block=False):
self._check_connected()
timeout = self.gettimeout()
try:
if timeout == 0.0 and block:
self.settimeout(None)
> self._sslobj.do_handshake()
E ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Basic Constraints of CA cert not marked critical (_ssl.c:1020)
/usr/lib64/python3.13/ssl.py:1363: SSLCertVerificationError
Anything else we need to know?:
Environment:
- Dask version: 2024.6.0
- Python version: 3.13
- Operating System: Fedora Rawhide
- Install method (conda, pip, source): distro packages (I am a distro packager working on rebuilds for Python 3.13, including getting distributed packaged, which means getting the tests to pass)