Airflow development workflow and DAG Deployment from feature branches

[nilskre]

Advice needed how to use Airflow in an enterprise setup.

Current setup

We have a monorepo for DAGs and two Airflow instances:

• prod -> DAGs from the main branch
• dev -> DAGs from the dev branch

Deployment uses the Git-Sync sidecar, which works fine.

New DAGs are developed (and tested with unit tests) on feature branches.
Some issues only become visible, when running the tasks on Airflow dev/prod. Therefore before merging the feature branch into dev/main, we want to test the DAG on dev/prod.

Challenge

Before merging a feature branch into dev or main, we want to test the new DAG on the corresponding Airflow instance.

Goal: Deploy DAGs from a feature branch to dev or prod for pre-merge testing without affecting the main DAGs.

Question

What is the recommended workflow to achieve this in an enterprise Airflow setup?

• Can we deploy feature-branch DAGs safely to dev/prod?
• Are there best practices or patterns for testing feature branches on live Airflow instances?
• Any tooling, Git workflows, or Airflow configurations that make this easier?
• Any advice or references to best practices would be highly appreciated.

Thanks in advance!

[mohhddhassan]

Yes, you can safely deploy feature-branch DAGs to dev/prod — but use namespaced DAG IDs to avoid collisions.

Best practice: feature DAGs → dev cluster (isolated from prod) → ephemeral DAG IDs → tested → merge → promoted.

For large orgs, ephemeral Airflow per branch is the cleanest but more infra-heavy.

[mohhddhassan]

Your welcome ;),
Git-Sync only supports one branch, so you can’t directly pull both dev and feature branches at the same time.
Option 1: Run multiple Git-Sync sidecars, each syncing a different branch into separate folders.

Option 2: Keep Git-Sync on dev, and let CI/CD copy feature DAGs (with namespaced DAG IDs) into a feature_dags/ folder inside dev.

Option 3: Spin up ephemeral Airflow environments per feature branch, with Git-Sync pointing to that branch.

[nilskre]

Thanks a lot!
Options 1 and 2 seem like a good fit for my use case.
For the actual implementation, could you maybe point me to some online resources or examples where this approach is explained or demonstrated in detail?

[mohhddhassan]

For Option 1, check the git-sync docs
and this GitHub issue on running multiple sidecars.
For Option 2, Astronomer’s CI/CD guide and their best practices
cover branch-based deployments and feature DAG workflows.

[potiuk]

Also in Airflow 3 you have bundles and specifically Git Bundles that Airflow natively supports without GitSync. My suggestion is to look at those.

[mohhddhassan]

Thanks - that's great suggestion.

[nilskre]

Finally had the time to come back to this. Thanks a lot for the links and suggestions!

It took me some time to understand the context and the idea/implementation of GitBundles.
Finally realized that GitBundles are a great feature and are exactly what we need in our use case.

Through GitBundles, we have been able to implement our own custom GitBundle. This custom GitDagBundle only contains changed DAGs from feature branches. The dag_id and dag_display_name are prefixed by the branch name to avoid collisions.

Nevertheless from my own experience coming up with this solution was quite challenging.

• In the beginning I was a little bit overwhelmed how everything works together and what are possible and clean ways to solve our problem. I think our setup with main/dev/feature branches the git-sync sidecar and the need to test DAGs on prod/dev prior to merging is a quite typical setup. For me a hands-on guide, explaining step by step how this could be achieved and which Airflow features are used for this, could have helped a lot. Would you see something like this as part of the Airflow docs? Or is this something which should be outside of the core Airflow docs?

• Shouldn't the DAGBundle functionality be mentioned on this docs page as well?

• The current GitDagBundle supports getting all DAGs from one repository. To make it much easier for others with the same use case: Maybe such a FeatureBranchGitDagBundle could also be part of the airflow git provider?

Happy about your thoughts on this. @potiuk
Thanks in advance!