`@task.kubernetes` fails on non-root images due to `/airflow/xcom` directory creation

[jhgoebbert]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

The @task.kubernetes decorator fails when used with non-root container images such as the official apache/airflow image.

After PR #28942, the initialization phase for Kubernetes-decorated tasks creates the directory /airflow/xcom (see providers/cncf/kubernetes/decorators/kubernetes.py#L98).

However, in the default Airflow image the container runs as the airflow user, (see HERE) which does not have write permissions to /. As a result, the attempt to create /airflow/xcom fails with a permissions error, and the Kubernetes task cannot start.

{"timestamp":"2025-10-10T13:10:06.622360Z","level":"info","event":"[base] + mkdir -p /airflow/xcom","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-10T13:10:06.622493Z","level":"info","event":"[base] mkdir: cannot create directory ‘/airflow’: Permission denied\n"

What you think should happen instead?

The Kubernetes-decorated task should run successfully, even when the image runs as a non-root user.

How to reproduce

1. Use the official Apache Airflow 3 image, e.g.:

   image: apache/airflow:3.1.0

2. Define a DAG:

    from airflow.decorators import dag, task
    from datetime import datetime
   
    @dag(start_date=datetime(2024, 1, 1), schedule=None)
    def test_dag():
    	@task.kubernetes(image="apache/airflow:3.1.0")
    	def example_task():
    		print("Hello from Airflow task")
   
    	example_task()
   
    test_dag()

3. Trigger the DAG.

Operating System

Kubernetes (Airflow charts)

Versions of Apache Airflow Providers

Latest

Deployment

Official Apache Airflow Helm Chart

Deployment details

Airflow version: 3.1.0
Executor: @task.kubernetes)
Base image: apache/airflow:3.1.0
Kubernetes version: (v1.32)

Anything else?

The /airflow/xcom directory is currently created unconditionally during initialization, assuming root permissions.
Possible approaches to resolve this:

• Create the directory under a writable path such as /tmp/airflow/xcom, or
• Make the XCom path configurable for Kubernetes-decorated tasks, or
• Check for existence and create the directory only if it’s writable by the current user.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

The change you mention was released in 5.2.1 more than 2 years and half ago, I'm surprised no one faced/reported the issue before. This would make sense if we have the folder /airflow created in the Docker image and we removed it recently, but according the blame history, it is not the case:

 git blame Dockerfile | grep "/airflow"
07fd0d71c8f (Jarek Potiuk       2020-04-02 19:52:11 +0200   43) ARG AIRFLOW_HOME=/opt/airflow
3feb057f0ee (Jarek Potiuk       2022-01-08 20:41:29 +0100   46) ARG AIRFLOW_USER_HOME_DIR=/home/airflow
94b03f6f432 (Jarek Potiuk       2021-08-04 23:32:12 +0200   62) ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
7f02b4718bc (Kamil Breguła      2022-01-23 14:15:45 +0100   63) ARG AIRFLOW_IMAGE_README_URL="https://raw.githubusercontent.com/apache/airflow/main/docs/docker-stack/README.md"
a1717a652b3 (Jarek Potiuk       2024-03-06 01:27:15 +0100   70) # SOURCES_FROM/TO with "." and "/opt/airflow" respectively - so that sources of Airflow (and all providers)
14b00ed5e2c (Josef Šimánek      2025-06-22 11:57:12 +0200  892)     echo "${COLOR_YELLOW}See: https://airflow.apache.org/docs/docker-stack/build.html#adding-new-pypi-packages-individually${COLOR_RESET}"
d4473555c0e (Jarek Potiuk       2025-04-02 13:11:13 +0200 1116)               --editable ./airflow-core --editable ./task-sdk --editable ./airflow-ctl \
09544966741 (GPK                2025-09-11 20:37:02 +0100 1119)               --editable ./airflow-e2e-tests \
9cbab95fd29 (Jarek Potiuk       2022-03-27 19:19:02 +0200 1474)     # HOME directory to the /home/airflow so that (for example) the ${HOME}/.local folder where airflow is
14b00ed5e2c (Josef Šimánek      2025-06-22 11:57:12 +0200 1544)         >&2 echo "     https://airflow.apache.org/docs/docker-stack/entrypoint.html"
14b00ed5e2c (Josef Šimánek      2025-06-22 11:57:12 +0200 1558)         >&2 echo "     https://airflow.apache.org/docs/docker-stack/entrypoint.html"
9cbab95fd29 (Jarek Potiuk       2022-03-27 19:19:02 +0200 1601)     >&2 echo "         https://airflow.apache.org/docs/docker-stack/build.html"
9cbab95fd29 (Jarek Potiuk       2022-03-27 19:19:02 +0200 1638) readonly DIRECTORY="${AIRFLOW_HOME:-/usr/local/airflow}"
9cbab95fd29 (Jarek Potiuk       2022-03-27 19:19:02 +0200 1663) # The content below is automatically copied from scripts/docker/airflow-scheduler-autorestart.sh
25b62a72f44 (Jarek Potiuk       2022-03-31 13:39:46 +0200 1664) COPY <<"EOF" /airflow-scheduler-autorestart.sh
9cbab95fd29 (Jarek Potiuk       2022-03-27 19:19:02 +0200 1672)         date >> /tmp/airflow_scheduler_errors.txt
7c12a9d4e0b (Jarek Potiuk       2020-06-16 12:36:46 +0200 1756) ARG AIRFLOW_REPO=apache/airflow
d524cec99db (Jarek Potiuk       2021-02-21 18:56:55 +0100 1761) ARG CONSTRAINTS_GITHUB_REPOSITORY="apache/airflow"
acff1296886 (Jarek Potiuk       2022-07-20 19:30:03 +0200 1778) # set to "." and "/opt/airflow" respectively.
8e3d1fcff6c (Jarek Potiuk       2022-03-29 15:03:50 +0200 2015) COPY --from=scripts airflow-scheduler-autorestart.sh /airflow-scheduler-autorestart
cf510a30fb4 (Jarek Potiuk       2020-06-27 14:29:55 +0200 2018) # See https://github.com/apache/airflow/issues/9248
4620770af45 (Jarek Potiuk       2022-01-11 10:38:34 +0100 2037) # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
2c6c7fdb230 (Jarek Potiuk       2021-01-21 16:16:09 +0100 2082)   org.opencontainers.image.url="https://airflow.apache.org" \
94b03f6f432 (Jarek Potiuk       2021-08-04 23:32:12 +0200 2083)   org.opencontainers.image.documentation="https://airflow.apache.org/docs/docker-stack/index.html" \

Could you try reproduce the issue in the K8S integration tests? That way we can figure out where and how we should fix it.

[hussein-awala]

Make the XCom path configurable for Kubernetes-decorated tasks, or

This should be the fastest and easiest solution to implement (it's full b/c), feel free to open a PR.

[jhgoebbert]

You can see from the tests of the kubernetes provider that only images with an entrypoint as root are used:

• ubuntu:16.04
• alpine
• bash:5.2
• perl:5.34.0
• gcr.io/spark-operator/spark:v2.4.4

So the integration tests cannot show this issue.

---

I have added the links for each image to its layer-details to show the root-style of the image
and the link to the tests where it is used in the kubernetes-provider below.

image="ubuntu:16.04"

• example_kubernetes.py#L62
• example_kubernetes.py#L114
• example_kubernetes_async.py#L62
• example_kubernetes_async.py#L115
• example_kubernetes_async.py#L140

image="alpine"

• example_kubernetes.py#L152
• example_kubernetes_async.py#L183

image="bash:5.2"

• example_kubernetes_cmd_decorator.py#L41

image="perl:5.34.0"

• example_kubernetes_job.py#L50C9-L50C28
• example_kubernetes_job.py#L69
• example_kubernetes_kueue.py#L118

image: "gcr.io/spark-operator/spark:v2.4.4"

• example_spark_kubernetes_spark_pi.yaml#L27

[jhgoebbert]

Could it be that it is intended that non-root images have no support?
I might miss something here - but I do not see a reason why 🤔

[hussein-awala]

In fact, it’s recommended to use rootless mode, which means this is a bug.

[jhgoebbert]

@hussein-awala Is there any additional information I can give or can you remove the pending-response tag?