SSL: CERTIFICATE_VERIFY_FAILED in airflow-worker for certificates with internal CA

[frodo2000]

Apache Airflow version

3.0.6

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When API server has setup SSL certificate with local Certificate Authority, airflow-worker is not enable to finish task and service log contains the following error:

Sep 11 13:13:06 dwh-airflow-dev bash[10802]: [2025-09-11 13:13:06 +0000] [10802] [INFO] Handling signal: term Sep 11 13:13:06 dwh-airflow-dev bash[10804]: [2025-09-11 13:13:06 +0000] [10804] [INFO] Worker exiting (pid: 10804) ...skipping... Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ) │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ do = <tenacity.DoAttempt object at 0x7dc38a77b2f0> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = { │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content': │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ '{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflo… │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'follow_redirects': <httpx._client.UseClientDefault object at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1 │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ } │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry_state = <RetryCallState 138278795187296: attempt #5; slept for 6.92; last result: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed (ConnectError [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ failed: unable to get local issuer certificate (_ssl.c:1000))> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <Retrying object at 0x7dc390fe3ad0 (stop=<tenacity.stop.stop_after_attempt │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ object at 0x7dc38a743020>, wait=<retryhttp._wait.wait_context_aware object at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc38a742ff0>, sleep=<function sleep at 0x7dc38a8e6b60>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ retry=<tenacity.retry.retry_any object at 0x7dc38a742fc0>, before=<function │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ before_nothing at 0x7dc38a8e7d80>, after=<function after_nothing at │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 0x7dc38a8e6ac0>)> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/airflow/sdk/api/client.py:735 in request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ❱ 735 return super().request(*args, **kwargs) │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ args = ('PATCH', 'task-instances/01993cd3-9124-7f15-9d8a-9f86b86d6c2d/run') │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ kwargs = { │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'content': │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ '{"state":"running","hostname":"dwh-airflow-dev.xxx.xx","unixname":"airflow","'… │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'data': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'files': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'json': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'params': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'headers': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'cookies': None, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'auth': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'follow_redirects': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ 'timeout': <httpx._client.UseClientDefault object at 0x7dc398afe090>, │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ ... +1 │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ } │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ self = <airflow.sdk.api.client.Client object at 0x7dc38a8517c0> │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:825 in request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:914 in send │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:942 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_auth │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:979 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_handling_redirects │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_client.py:1014 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ _send_single_request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:249 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ handle_request │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /usr/lib/python3.12/contextlib.py:158 in __exit__ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ /home/airflow/airflow_venv/lib/python3.12/site-packages/httpx/_transports/default.py:118 in │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: │ map_httpcore_exceptions │ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ Sep 12 07:28:54 dwh-airflow-dev bash[27220]: ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer Sep 12 07:28:54 dwh-airflow-dev bash[27220]: certificate (_ssl.c:1000)

In SDK Client class I found that only server certificate is added to Certifi library:
ctx = ssl.create_default_context(cafile=certifi.where()) if API_SSL_CERT_PATH: ctx.load_verify_locations(API_SSL_CERT_PATH)

Then only self-signed certificates are valid but certificates with local CA (or even intermediate local CA) becomes invalid.
We should consider add additional config parameter like API_SSL_CA_BUNDLE_PATH and add it to certifi context.

Checked workaround:

• adding CA certificated directly to certifi/cacert.pem file - airflow-worker works correctly but each certifi update needs to cacert.pem redefinition
• adding env variables REQUESTS_CA_BUNDLE or SSL_CERT_FILE doesn't work

What you think should happen instead?

No response

How to reproduce

Create SSL certificate with local CA chain

Operating System

Ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

Systemd services for:

• postgresql
• redis
• airflow-api
• ariflow-scheduler
• airflow-dag-processor
• airflow-triggerer
• airflow-worker
  with env configuration in /etc/airflow.cfg file
  Celery Executor used

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[emredjan]

Same problem I am having here: #55147

[potiuk]

It seems that what should work for you is to make sure your self-signed certificate is configured as ssl_cert #55147 also - ability to add trust store option has been discussed there, PRs are welcome if that is a better option for you.

Converting to a discussion