urllib certificate verify issues with pyenv on Python 3.10.7, CentOS 7, Alt OpenSSL 1.1.1

[bwfisher82]

Steps Taken

We have successfully installed Python 3.10.7 on CentOS 7 via pyenv after days of trying.

We are using the latest stable release of OpenSSL 1.1.1 which we compiled from source and stuck at /usr/local/ssl.

We are then able to install Python 3.10.7 via pyenv with the following option set:

export CONFIGURE_OPTS="--with-openssl=/usr/local/ssl --with-openssl-rpath=auto --with-ssl-default-suites=openssl"

This worked! Finally! After days of research.

Error with urllib

However, when we test urllib to a SSL site, regardless of the site, we have urllib errors "CERTIFICATE_VERIFY_FAILED".

As a note this error occurs on every version of Python I've tested with not just the 3.10.7 we're trying to use. 3.6.6, 3.7.14, 3.8.14, 3.9.14, 3.10.7 all compiled using the CONFIGURE_OPTS above. So it would seem to not be a Python issue and how it is being installed.

OpenSSL Version

$ python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
3.10.7 (main, Oct  4 2022, 18:05:59) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
OpenSSL 1.1.1r-dev  xx XXX xxxx

Urllib test

$ python -c "from urllib.request import urlopen; urlopen('https://www.python.org')"
<truncated>
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

Detailed Steps

OpenSSL 1.1.1

• install dependencies
• clone openssl repo, checkout OpenSSL_1_1_1-stable
• Ran the below.

./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
make && make test && sudo make install
./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo ldconfig -v

Python 3.10.7 via pyenv:

• standard pyenv install for Linux
• add the PYENV_ROOT etc to .bashrc
• Ran the below.

export CONFIGURE_OPTS="--with-openssl=/usr/local/ssl --with-openssl-rpath=auto --with-ssl-default-suites=openssl"
pyenv update
pyenv install 3.10.7

[native-api]

Did you import root certificates into your custom OpenSSL installation? Each OpenSSL installation can have its own store (this is probably a configure option with the default being somewhere under installation prefix). https://stackoverflow.com/questions/4138139/how-to-find-out-the-path-for-openssl-trusted-certificates

[bwfisher82]

Okay, yeah, I don't see anything in openssl compile from source that has an option for this, but if you dig deep into the docs I see a comment that root/certs and root/private are empty initially and other cert files don't exist. On RH based systems it looks like I needed to link over to files in /etc/pki/tls.

Specifically this fixed it, I think:

cd /usr/local/ssl
sudo ln -s /etc/pki/tls/certs certs
sudo ln -s /etc/pki/tls/private private
sudo ln -s /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem cert.pem

Specifically, it still didn't work without that last one.

Thanks.