Airflow Deployment on K8s Cluster

[ChakshuGautam]

Problem Statement

Setup Airflow on a Kubernetes Cluster

Success Criteria

1. Update the Airflow docker custom image to latest version.
2. Setting up Kubernetes Executor
3. Ability to load DAGs from the common-dags repo. Github CI/CD to be setup to update the dags whenever code is pushed to specific branches.
4. Design a way to upgrade Airflow without loss of DAG state.
5. Scaling up and down the executor based on jobs. You can use sample jobs for this.
6. Ability to load Hashicorp Vault Credentials in the DAGs.
7. Deploy Hashicorp Vault as part of the installation. [Good to have]
8. Setting up two instances - dev and production. [Good to have]

Links

1. The docker image for Airflow to be used
2. DAGs
3. Hashicorp Vault Airflow Config

[yuvraj9]

Scaling up and down the executor based on jobs. You can use sample jobs for this.

For this one what metric do you want to use. Does default CPU/Memory based scaling works?

[ChakshuGautam]

Scaling up and down the executor based on jobs. You can use sample jobs for this.

For this one what metric do you want to use. Does default CPU/Memory based scaling works?

Hey @yuvraj9 yes you can use the CPU 80/20 kind of a thing. 80 => +1 20 => -1
Are you setting up HPA for this?

[yuvraj9]

Hey Chakshu, I will setup HPA and will do some load testing for this

[yuvraj9]

Hey @ChakshuGautam could you give me access to this -> https://github.com/Samagra-Development/airflow

[ChakshuGautam]

Added @yuvraj9 please check.