Best practice for webserver liveness probe check

[hanxdatadog]

Description

👋 Dear Airflow community,

Recently we ran some stress tests on Airflow’s asset-based scheduling and noticed that the webserver was frequently restarting due to liveness probe failures. The liveness probe we were using was:

/api/v2/monitor/health

This was based on the guidance from the old health endpoint response:

airflow/airflow-core/src/airflow/api_fastapi/core_api/app.py

Line 85 in 31f0eac

content={"error": "Moved in Airflow 3. Please change config to check `/api/v2/monitor/health`"},

From reading the source code, my understanding is that /api/v2/monitor/health checks the overall health of the metadatabase, scheduler, and triggerer. If there’s any slowdown in retrieving health information from these components, the webserver gets restarted, which makes the UI unavailable. Ideally, we’d like the UI to remain available even if the metadb or scheduler is under heavy load.

What would be the recommended alternative liveness check that doesn’t make the webserver’s health dependent on backend components? I see some options, such as the execution API health endpoint:

airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/health.py

Line 30 in 31f0eac

return {"status": "healthy"}

I also noticed that the official chart for the API server uses the version endpoint:

airflow/chart/templates/api-server/api-server-deployment.yaml

Line 194 in 31f0eac

path: /api/v2/version

Any suggestions or guidance would be much appreciated 🙏

Use case/motivation

A liveness probe check API end point for webserver that is not dependent on other components

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Please use discussions for questions, not issues (as instructed in the template). Converted it.

[potiuk]

I guess our Helm chart is a good one to look at.

[brokenjacobs]

We still run into failures of the liveness probe even with /api/v2/version in the check. I've tried upping timeouts in the liveness probe and it doesn't help. Any time we have terraform hit the api endpoint to check stored variables for instance, it will 500 and restart due to liveness failures.

[potiuk]

You should look what kind of failures they are. Maybe your liveness probe does not have enough memory for example? It's all in your logs that you need to look at.

[brokenjacobs]

This is a kubernetes deployment. The liveness probe comes from kubelet. The kubernetes events (not logs) indicate a timeout checking for liveness, and the pod eventually receives a SIGTERM (which is in the api server log) from kubelet because liveness fails. There is no OOM kill. OOM Killer sends SIGKILL.

Airflow 2 did not have issues with liveness when the api was lightly used. This is something new with airflow3. I’ve extended the timeouts and retries for liveness, and still have the same issues. I’ve increased pod resources (more cpu/ram) and even switched to a faster instance type for running the pod. Still having these issues.

[potiuk]

Then I suggest to open an issue specifically for Airflow 3 and possibly mention the timeouts you experience -> specify the exact liveness probe you use and dumping there some logs from run liveness probes. It would be interesting likely to see if the liveness probes are usually fast, but timeout sometimes, or whether they consistently take a lot of time, and sometimes just reach the threshold.

Opening an issue in this case is likely best thing you can do - we are going (As a community https://lists.apache.org/thread/8qymxm5q39gdorhl8tccz9f3qo92llm7 ) on polishing and fixing remaining issues in Airflow 3.1 for the next few weeks, and clear and well diagnosed cases from our users might help in solving the issues faster.

[brokenjacobs]

Thanks for responding!

I managed to track this down to database pool size issues. I was only seeing the timeouts when hitting the api server with terraform to set up pools/variables, etc. And terraform uses parallel connections to do this. For whatever reason this wasn't an issue on airflow 2.x but on 3.x I've had to increase the number of connections I use to the db server. We use pg_bouncer, so I had plenty available, but for whatever reason the api server was timing out on getting a new connection to the DB with the default settings.

I just double checked. Specifically this was pgBouncer settings in the helm chart:
pgbouncer:
metadataPoolSize: 40
resultBackendPoolSize: 20

It wasn't an apiserver setting but a pgbouncer setting. We need more connections with airflow3 is the tldr;