Skip to content

13_airflow_dependencies_and_extras.rst

Latest commit

 

History

History
430 lines (324 loc) · 25.8 KB

13_airflow_dependencies_and_extras.rst

File metadata and controls

430 lines (324 loc) · 25.8 KB

Airflow dependencies

This document describes how we manage Apache Airflow dependencies, as well as how we make sure that users can use Airflow both as an application and as a library when they are deploying their own Airflow - using our constraints mechanism.

Airflow pyproject.toml files and uv workspace

Managing dependencies is an important part of developing Apache Airflow, we have more than 700 dependencies, and often when you add a new feature that requires a new dependency, you should also update the dependency list. This also happens when you want to add a new tool that is used for development, testing, or building the documentation and this document describes how to do it.

When it comes to defining dependencies for any of the Apache Airflow distributions, we are following the standard pyproject.toml format that has been defined in PEP 518 and PEP 621, we are also using dependency groups defined in PEP 735 - particularly dev dependency group which we use in all our pyproject.toml files to define development dependencies.

We have big number (currently more than 100) python distributions in Apache Airflow repository - including the main apache-airflow package, apache-airflow-providers-* packages, apache-airflow-core package, apache-airflow-task-sdk package, apache-airflow-ctl package, and several other packages.

They are all connected together via uv workspace feature (workspace is defined in the root pyproject.toml file of the repository in the apache-airflow distribution definition). The workspace feature allows us to run uv sync at the top of the repository to install all packages in editable mode in the development environment from all the distributions and resolve the dependencies together, so that we know that the dependencies have no conflicting requirements. Also the distributions are referring to each other via name - which means that when you run locally uv sync, the local version of the packages are used, not the ones released on PyPI, which means that you can develop and test changes that span multiple packages at the same time. This is a very powerful feature that allows us to maintain the complex ecosystem of Apache Airflow distributions in a single monorepo, and allows us - for example to add new feature to common distributions used by multiple providers and test them all together before releasing new versions of either of those packages.

Managing dependencies in pyproject.toml files

Each of the pyproject.toml files in Apache Airflow repository defines dependencies in one of the following sections:

  • [project.dependencies] - this section defines the required dependencies of the package. These dependencies are installed when you install the package without any extras.
  • [project.optional-dependencies] - this section defines optional dependencies (extras) of the package. These dependencies are installed when you install the package with extras - for example pip install apache-airflow[ssh] will install the ssh extra dependencies defined in this section.
  • [dependency-group.dev] - this section defines development dependencies of the package. These dependencies are installed when you run uv sync by default. when uv syncs sources with local pyproject.toml it adds dev dependency group and package is installed in editable mode with development dependencies.

Adding and modifying dependencies

Adding and modifying dependencies in Apache Airflow is done by modifying the appropriate pyproject.toml file in the appropriate distribution.

When you add a new dependency, you should make sure that:

  • The dependency is added to the right section (main dependencies, optional dependencies, or development dependencies)

  • Some parts of those dependencies might be automatically generated (and overwritten) by our prek hooks. Those are the necessary dependencies that prek hooks can figure out automatically by analyzing the imports in the sources and structure of the project. We also have special case of shared dependencies (described in shared dependencies document) where we effectively "static-link" some libraries into multiple distributions, to avoid unnecessary coupling and circular dependencies, and those dependencies contribute some dependencies automatically as well. Pay attention to comments such us example start and end of such generated block below. All the dependencies between those comments are automatically generated and you should not modify them manually. You might instead modify the root source of those dependencies - depending on the automation, you can usually also add extra dependencies manually outside of those comment blocks

    # Automatically generated airflow optional dependencies (update_airflow_pyproject_toml.py)
# ....
# LOTS OF GENERATED DEPENDENCIES HERE
# ....
# End of automatically generated airflow optional dependencies

  • The version specifier is as open as possible (upper) while still allowing the package to install and pass all tests. We very rarely upper-bind dependencies - only when there is a known conflict with a new or upcoming version of a dependency that breaks the installation or tests (and we always make a comment why we are upper-bounding a dependency).

  • Make sure to lower-bind any dependency you add. Usually we lower-bind dependencies to the minimum version that is required for the package to work but in order to simplify the work of resolvers such as pip, we often lower-bind to higher (and newer) version than the absolute minimum especially when the minimum version is very old. This is for example good practice in boto and related packages, where new version of those packages are released frequently (almost daily) and there are many versions that need to be considered by the resolver if the version is not new enough.

  • Make sure to run uv sync after modifying dependencies to make sure that there are no conflicts between dependencies of different packages in the workspace. You can run it in multiple ways - either from the root of the repository (which will sync all packages) or from the package you modified (which will sync only that package and its dependencies). Also good idea might be to run uv sync --all-packages --all-extras at the root of the repository to make sure that all packages with all extras can be installed together without conflicts, but this might be sometimes difficult and slow as some of the extras require some additional system level dependencies to be installed (for example mysql or postgres extras require client libraries to be installed on the system).

  • Make sure to run all tests after modifying dependencies to make sure that nothing is broken

Referring to other Apache Airflow distributions in dependencies

With having more than distributions in the repository, it is often necessary to refer to other distributions in the dependencies in order to use some common features or simply to use the features that the other distribution provides. There are two ways of doing it:

  • Regular package linking with apache-airflow-* dependency
  • Airflow "shared dependencies" mechanism - which is a bit of a custom hack for Airflow monorepo that allows us to "static-link" some common dependencies into multiple distributions without creating circular dependencies.

We are not going to describe the shared dependencies mechanism here, please refer to the shared dependencies document for details, but there are certain rules when it comes to referring to other Airflow distributions in dependencies - here are the important rules to remember:

  • You can refer to other distributions in your dependencies - as usual using distribution name. For example, if you are adding a dependency to apache-airflow-providers-common-compat package from apache-airflow-providers-google, you can just add apache-airflow-providers-common>=x.y.z to the dependencies and when you run uv sync, the local version of the package will be used automatically (this is thanks to the workspace feature of uv that does great job of binding our monorepo together). Some of those are added automatically by prek hooks - when it can detect such dependencies by analyzing imports in the sources - then they are added automatically between the special comments mentioned above, but sometimes (especially when such dependencies are not at the top-level imports) you might need to add them manually.

  • In case you make a feature change in a distribution and would like to update its version, you should never update the distribution version on your own. It is entirely up to the Release Manager to bump the version of distributions that are defined as project.version. This goes almost without an exception and any diversions from this rule should be discussed at #release-management channel in Airflow Slack beforehand. The only exception to this rule is when you are adding a new distribution to the repository - in that case you should set the initial version of the distribution - usually 0.0.1, 0.1.0 or 1.0.0 depending on the maturity of the package. But still you should discuss it in the channel.

  • Sometimes, when you add a new feature to a common distribution, you might add a feature to it or change the API in the way that other packages can use it. This is especially true for common packages such as apache-airflow-providers-common-compat, but can happen for other packages (for example apache-airflow-providers-apache-beam is used by apache-airflow-providers-google to use Apache Beam hooks to communicate with Google Dataflow). In such case, when you are adding a feature to a common package remember that the feature you just add will only be released in the FUTURE release of such common package and you cannot add >==x.y.z dependency to it where x.y.z is the version you are going to release in the future. Ultimately, this should happen (and happens) when the Release Manager prepares both packages together. Let us repeat - such changes in versions between different airflow package should NOT be added to the dependencies manually by the contributor. They should exclusively be added by the Release Manager. when preparing the release of both packages together. We have a custom mechanism to support such additions, where it is contributor's responsibility to mark dependency with a special comment - simply communicating with the Release Manager that such dependency should be updated to the next version when the release is prepared. If you see such a need to use newly added feature and using it at the same time in a different distribution - make sure to add this comment in the line where dependency you want to use the new feature from is defined:

    Warning

    You must use the exact comment # use next version - otherwise the automation will not pick it up.

    # ...
"apache-airflow-SOMETHING>1.0.0",  # use next version
# ...

    We have tooling in place to:

    • check that no regular PRs modify such cross-dependency versions
    • make sure that such dependencies marked with the comment are updated automatically to the next version, when preparing the release

    Example of such dependency (for example placed in apache-airflow-providers-google/pyproject.toml) when you want to use a new feature you are adding in the same PR to apache-airflow-providers-common-compat package:

    "apache-airflow-providers-google>=1.2.0",
"apache-airflow-providers-common-compat>=5.5.0",  # use next version
"requests>=2.25.1",

    Note that you SHOULD NOT change version of the common-compat package in this PR. When the Release Manager will prepare the release of both packages, the version will be updated automatically to the next version that is being released (for example 5.6.0) and the Release Manager will make sure that both packages are released together.

  • Some cross dependencies like that are added automatically by prek hooks as well - when it can detect such dependencies by analyzing imports in the sources - then they are added automatically between the special comments mentioned above. In such case the dependency line will be added in the section that is commented around with # Start automated ..., # End automated ... comments as described above. In this case, just copy the dependency line to outside of those comments and add the # use next version comment to it, next time when prek hook will be run it will remove the automatically added line and keep only the manually added line with the comment.

# There is an automated check for that in case of common.compat provider - because it is one that changes
often and almost always when it changes, there are some providers that need those changes - so you should add the # use next version comment to such providers always when you modify common.compat. In case there is a change in the common.compat package that requires updating other providers, and there are no changes in other providers that require such update - you will get and error in the Selective Check CI job that will remind you to add such comments to the providers that need to be updated. You can skip the check if you are sure that no other providers do not need to be updated by adding skip common compat check label to the PR. Only maintainers and collaborators can add such label.

  • Some of our dependencies have forced minimum version - mostly because of the Airflow 3 minimum version compatibility. Just in case in the future, we have other distributions referring to them we are forcing a minimum version for those distributions by a prek hook. This causes entries like this:

    "apache-airflow-providers-google-something>=1.2.0",
"apache-airflow-providers-fab>=2.2.0",  # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"requests>=2.25.1",

    This will not happen, when the distribution will depend on even newer version of the dependency, but this is mainly a precaution in cases where we know we need a minimum version for Airflow 3 compatibility (for example for git, common.messaging and few other providers) or where we know that we should not use older version of providers in the future because some functionality in them stopped working (like in case of amazon, fab). You are free to modify those versions to higher versions if you need to, and prek will remove those comments automatically.

Our CI system will do all the tests for you anyway - including running some lower-bind checks on dependencies. For example it will take each provider in a turn and will try to resolve lowest-possible dependencies defined for that provider and see if the tests are still passing, so we should be relatively protected against putting too low lower-bounds on dependencies, but running uv sync locally is still a good idea to find such things before they hit the CI system.

Airflow as both library and application - constraint files

Why constraints?

Airflow is not a standard python project. Most of the python projects fall into one of two types - application or library. As described in this StackOverflow question, the decision whether to pin (freeze) dependency versions for a python project depends on the type. For applications, dependencies should be pinned, but for libraries, they should be open.

For applications, pinning the dependencies makes it more stable to install in the future - because new (even transitive) dependencies might cause installation to fail. For libraries - the dependencies should be open to allow several different libraries with the same requirements to be installed at the same time.

The problem is that Apache Airflow is a bit of both - application to install and library to be used when you are developing your own operators and Dags.

This - seemingly unsolvable - puzzle is solved by having pinned constraints files.

Why not using a standard approach?

Because the standards defined in Python environment have not yet caught up with the needs of complex projects that are both libraries and applications.

What we do is more of a hack to overcome the limitations of existing tools, when it comes to reproducible installations of such projects.

Discussion about standards advanced over the last few years with PEP 751 that introduces A file format to record Python dependencies for installation reproducibility, but support for this format (pylock.toml) is not yet widespread. As of November 2025 it is experimental in pip and supported as export format for uv from their own lock (development environment focused) file. But the ability to use the format for reproducible installation as intended in the PEP 751 is not yet supported, not even PEP 751 is complete enough to allow that.

The discussion is on-going on further advancing this mechanism with follow-up PEP, to support such reproducible installation process that Airflow introduced years ago with the constraints hack - see `Pre-PEP <https://discuss.python.org/t/pre-pep-add-ability-to-install-a-package-with-reproducible-dependencies/99497/14>_`_ where discussion about doing something like we do is on-going.

Pinned constraints files

Note

Only pip and uv installation is officially supported.

While it is possible to install Airflow with tools like poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported. Uv follows pip approach with uv pip so it should work similarly.

There are known issues with bazel that might lead to circular dependencies when using it to install Airflow. Please switch to pip if you encounter such problems. The Bazel community added support for cycles in this PR so it might be that newer versions of bazel will handle it.

If you wish to install Airflow using these tools you should use the constraint files and convert them to appropriate format and workflow that your tool requires.

By default when you install apache-airflow package - the dependencies are as open as possible while still allowing the apache-airflow package to install. This means that the apache-airflow package might fail to install when a direct or transitive dependency is released that breaks the installation. In that case, when installing apache-airflow, you might need to provide additional constraints (for example pip install apache-airflow==1.10.2 Werkzeug<1.0.0)

There are several sets of constraints we keep:

  • constraints - these are constraints generated by matching the current Airflow version from sources
    and providers that are installed from PyPI. Those are constraints used by the users who want to install Airflow with pip, they are named constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt.
  • constraints-source-providers - these are constraints generated by using providers installed from current sources. While adding new providers their dependencies might change, so this set of providers is the current set of the constraints for Airflow and providers from the current main sources. Those providers are used by CI system to keep "stable" set of constraints. They are named constraints-source-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt
  • constraints-no-providers - these are constraints generated from only Apache Airflow, without any providers. If you want to manage Airflow separately and then add providers individually, you can use them. Those constraints are named constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt.

The first two can be used as constraints file when installing Apache Airflow in a repeatable way. It can be done from the sources:

from the PyPI package:

pip install "apache-airflow[google,amazon,async]==3.0.0" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.10.txt"

The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without extras.

When you install Airflow from sources (in editable mode) you should use "constraints-source-providers" instead (this accounts for the case when some providers have not yet been released and have conflicting requirements).

pip install -e ".[devel]" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.10.txt"

This also works with extras - for example:

pip install ".[ssh]" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.10.txt"

There are different set of fixed constraint files for different python major/minor versions and you should use the right file for the right python version.

If you want to update just the Airflow dependencies, without paying attention to providers, you can do it using constraints-no-providers constraint files as well.

pip install . --upgrade \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-no-providers-3.10.txt"

These constraint files are generated from the uv.lock file committed in the repository, using uv export --frozen to convert the lock file into a flat list of pinned versions suitable for pip install --constraint. This means the constraint files always reflect the same dependency versions that uv sync installs for developers.

The constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt and constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt will be automatically regenerated by CI job every time after the pyproject.toml is updated and pushed if the tests are successful.

Note

Only pip and uv installation is currently officially supported.

While there are some successes with using other tools like poetry or pip-tools, they do not share the same workflow as pip - especially when it comes to constraint vs. requirements management. Installing via Poetry or pip-tools is not currently supported. Uv follows pip approach with uv pip so it should work similarly.

There are known issues with bazel that might lead to circular dependencies when using it to install Airflow. Please switch to pip if you encounter such problems. Bazel community works on fixing the problem in this PR so it might be that newer versions of bazel will handle it.

If you wish to install Airflow using these tools you should use the constraint files and convert them to appropriate format and workflow that your tool requires.

Optional dependencies (extras) of apache-airflow package

There are a number of extras that can be specified when installing Airflow. Those extras can be specified after the usual pip install - for example pip install -e.[ssh] for editable installation. Note that there are two kinds of extras - regular extras (used when you install airflow as a user, but in editable mode you can also install devel extras that are necessary if you want to run Airflow locally for testing and doc extras that install tools needed to build the documentation.

Note that some of those extras are only added in the meta-distribution apache-airflow, they are not defined in the airflow-core and any other packages. The dependencies to providers and copy of such extras from the apache-airflow package is done automatically by our prek hooks, similarly as creating extras for providers from the apache-airflow-providers-* packages in the workspace.

There are also some manually defined extras for optional features that are often used with Airflow.

You can read more about those extras in the extras reference.

You can now check how to update Airflow's metadata database if you need to update structure of the DB.