CDR (Centralized DAG Repository)

This repository acts as the central source of truth for DAGs developed across various projects at BC.

Purpose

Centralized Management: Hosts and manages all production-ready Airflow DAGs.
Repository Dispatch Trigger: Listens for repository dispatch events from other repositories to dynamically sync relevant DAGs.
S3 Integration: Pulls DAGs from a configured S3 bucket based on dispatch payload.
Airflow Sync: Git-synced with the Airflow production server running in Kubernetes, ensuring that the latest DAGs are always available and deployed.

Other repositories trigger a repository_dispatch event when a new DAG or DAG update is ready - this happens during a Github release.
This repository receives the dispatch, fetches the DAG(s) from a specified S3 bucket.
DAGs are placed in the dags folder, committed and pushed.
GitSync propagates the change to the Airflow production server running in a Kubernetes pod.

An S3 bucket with DAGs stored in project-specific paths.
A GitHub personal access token with workflow and repository access for allowing the Git bit to push to this repository.
A Github actions ECR Role to assume so that the CI can authenticate to the AWS.
Kubernetes-deployed Airflow setup with DAG GitSync enabled.