Magnum fails to authenticate with Keystone when using self-signed TLS certs. #1158
Description
Describe the bug
When deploying magnum with openstack-helm, the default manifests configure the magnum service to use publicURL and public interface when authenticating with Keystone. This works fine with Let’s Encrypt or other publicly trusted CAs, but fails when the Keystone public endpoint is using a self-signed TLS certificate. Since pods don’t mount the CA, which causes SSL verification failures.
To Reproduce
Steps to reproduce the behavior:
- Updated the required override config for magnum in
/etc/genestack/helm-configs/magnum/magnum-helm-overrides.yaml. - Run the Magnum deployment Script
/opt/genestack/bin/install-magnum.sh - Check that all the
magnumpods are running. - Run debug while validating the
magnumservice.
Expected behavior
Magnum should successfully authenticate with Keystone without any error, even if Keystone uses a self-signed CA.
Server (please complete the following information):
- OS: Ubuntu 22.04
Additional context
- Confirm all Magnum pods are running.
(genestack) root@controller-01:~# kubectl get pods -n openstack|grep magnum
magnum-api-747f4f54c5-ts6b6 1/1 Running 0 18m
magnum-api-747f4f54c5-z2v72 1/1 Running 0 18m
magnum-conductor-0 1/1 Running 0 18m
magnum-conductor-1 1/1 Running 0 18m
magnum-db-sync-96d26 0/1 Completed 0 18m
magnum-domain-ks-user-6bm2g 0/1 Completed 0 17m
magnum-ks-endpoints-jp9mz 0/3 Completed 0 18m
magnum-ks-service-ndk64 0/1 Completed 0 18m
magnum-ks-user-dcqk4 0/1 Completed 0 18m- Run debug while validating the Magnum service:
# kubectl --namespace openstack exec -ti openstack-admin-client -- \
openstack --os-interface internal coe cluster list --debugOutput truncated
REQ: curl -g -i -X GET http://magnum-api.openstack.svc.cluster.local:9511/v1/clusters ...
RESP: [503] {"title": "Service Unavailable", "error": {"type": "SSLError"}}
magnumclient.exceptions.InternalServerError: ERROR: Internal Error
command terminated with exit code 1- Observed error in magnum-api logs:
2025-08-27 10:32:46.624 8 INFO keystonemiddleware.auth_token [None req-da0bbcdd-6815-4fdf-b300-7ea567070e4c - - - - - -] A version other than v3 was requested: public
2025-08-27 10:32:46.894 8 CRITICAL magnum [None req-da0bbcdd-6815-4fdf-b300-7ea567070e4c - - - - - -] Unhandled error: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://keystone.dev.genestack.com/v3/auth/tokens: HTTPSConnectionPool(host='keystone.dev.genestack.com', port=443): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)')))
2025-08-27 10:32:46.894 8 ERROR magnum Traceback (most recent call last):
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/connectionpool.py", line 716, in urlopen
2025-08-27 10:32:46.894 8 ERROR magnum httplib_response = self._make_request(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/connectionpool.py", line 404, in _make_request
2025-08-27 10:32:46.894 8 ERROR magnum self._validate_conn(conn)
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
2025-08-27 10:32:46.894 8 ERROR magnum conn.connect()
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/connection.py", line 419, in connect
2025-08-27 10:32:46.894 8 ERROR magnum self.sock = ssl_wrap_socket(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket
2025-08-27 10:32:46.894 8 ERROR magnum ssl_sock = _ssl_wrap_socket_impl(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl
2025-08-27 10:32:46.894 8 ERROR magnum return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2025-08-27 10:32:46.894 8 ERROR magnum return self.sslsocket_class._create(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/usr/local/lib/python3.12/ssl.py", line 1041, in _create
2025-08-27 10:32:46.894 8 ERROR magnum self.do_handshake()
2025-08-27 10:32:46.894 8 ERROR magnum File "/usr/local/lib/python3.12/ssl.py", line 1319, in do_handshake
2025-08-27 10:32:46.894 8 ERROR magnum self._sslobj.do_handshake()
2025-08-27 10:32:46.894 8 ERROR magnum ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)
2025-08-27 10:32:46.894 8 ERROR magnum
2025-08-27 10:32:46.894 8 ERROR magnum During handling of the above exception, another exception occurred:
2025-08-27 10:32:46.894 8 ERROR magnum
2025-08-27 10:32:46.894 8 ERROR magnum Traceback (most recent call last):
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
2025-08-27 10:32:46.894 8 ERROR magnum resp = conn.urlopen(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/connectionpool.py", line 802, in urlopen
2025-08-27 10:32:46.894 8 ERROR magnum retries = retries.increment(
2025-08-27 10:32:46.894 8 ERROR magnum ^^^^^^^^^^^^^^^^^^
2025-08-27 10:32:46.894 8 ERROR magnum File "/var/lib/openstack/lib/python3.12/site-packages/urllib3/util/retry.py", line 594, in increment
2025-08-27 10:32:46.894 8 ERROR magnum raise MaxRetryError(_pool, url, error or ResponseError(cause))
2025-08-27 10:32:46.894 8 ERROR magnum urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='keystone.dev.genestack.com', port=443): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)')))- I notice that the endpoint_type is set to publicURL and interface is set to public in the default magnum overrides
https://github.com/rackerlabs/genestack/blob/main/base-helm-configs/magnum/magnum-helm-overrides.yaml#L84
conf:
logging:
logger_root:
handlers:
- stdout
level: INFO
magnum:
barbican_client:
endpoint_type: publicURL
region_name: RegionOne
cinder_client:
endpoint_type: publicURL
region_name: RegionOne
.....
glance_client:
api_version: 2
endpoint_type: publicURL
region_name: RegionOne
heat_client:
endpoint_type: publicURL
region_name: RegionOne
keystone_auth:
auth_section: keystone_authtoken
keystone_authtoken:
auth_type: password
auth_version: v3
interface: public- In my lab, I fixed the issue by overriding these values: endpoint_type to internal and interface to internal, and reinstalling the magnum.
REQ: curl -g -i -X GET http://magnum-api.openstack.svc.cluster.local:9511/v1/clusters -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: container-infra latest" -H "User-Agent: None" -H "X-Auth-Token: {SHA256}793d84af0808371dfa96f18eeacb072d4e5515e7968c092f9e48a0461e0a24b1"
Starting new HTTP connection (1): magnum-api.openstack.svc.cluster.local:9511
http://magnum-api.openstack.svc.cluster.local:9511 "GET /v1/clusters HTTP/1.1" 200 16
RESP: [200] Connection: close Content-Length: 16 Content-Type: application/json OpenStack-API-Maximum-Version: container-infra 1.11 OpenStack-API-Minimum-Version: container-infra 1.1 OpenStack-API-Version: container-infra 1.11 vary: OpenStack-API-Version x-openstack-request-id: req-1ff1bfe0-7dcb-4650-90d3-21b32189d523
RESP BODY: {"clusters": []}
GET call to container-infra for http://magnum-api.openstack.svc.cluster.local:9511/v1/clusters used request id req-1ff1bfe0-7dcb-4650-90d3-21b32189d523
clean_up ListCluster:
END return value: 0