airflow-ext

Meltano Airflow utility extension

Apache Airflow 2.7+ is required, adjustments made for Airflow 3.x compatibility

Example meltano.yml entry

  utilities:
  - name: airflow
    namespace: airflow
    pip_url: >
      git+https://github.com/meltano/edk.git@main
      git+https://github.com/monomeric/airflow-ext.git@main
      apache-airflow==3.2.0
      --constraint https://raw.githubusercontent.com/apache/airflow/constraints-3.2.0/constraints-no-providers-${MELTANO__PYTHON_VERSION}.txt
    executable: airflow_invoker
    commands:
      describe:
        executable: airflow_extension
        args: describe
      initialize:
        executable: airflow_extension
        args: initialize
      invoke:
        executable: airflow_extension
        args: invoke
    settings:
    - name: database.sql_alchemy_conn
      label: SQL Alchemy Connection
      value: sqlite:///$MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/airflow.db
      env: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
    - name: core.dags_folder
      label: DAGs Folder
      value: $MELTANO_PROJECT_ROOT/orchestrate/airflow/dags
      env: AIRFLOW__CORE__DAGS_FOLDER
    - name: core.plugins_folder
      label: Plugins Folder
      value: $MELTANO_PROJECT_ROOT/orchestrate/airflow/plugins
      env: AIRFLOW__CORE__PLUGINS_FOLDER
    - name: core.load_examples
      label: Load Examples
      value: false
      env: AIRFLOW__CORE__LOAD_EXAMPLES
    - name: core.dags_are_paused_at_creation
      label: Pause DAGs at Creation
      env: AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION
      value: false
    - name: webserver.web_server_port
      label: Webserver Port
      value: 8080
      env: AIRFLOW__API__PORT
      description: |
        The web and API server port, formerly set by AIRFLOW__WEBSERVER__WEB_SERVER_PORT in Airflow 2.x. If this variable
        remains unset with Airflow 3.x, configuration generation may fail as the airflow.cfg file catches debug logs through
        'stdout' upon creation by 'airflow config list --defaults'.
    - name: logging.base_log_folder
      label: Base Log Folder
      value: $MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/logs
      env: AIRFLOW__LOGGING__BASE_LOG_FOLDER
      description: |
        The folder where airflow should store its log files. This path must be absolute. There are a few existing
        configurations that assume this is set to the default. If you choose to override this you may need to update
        the dag_processor_manager_log_location and child_process_log_directory settings as well.
    - name: logging.dag_processor_manager_log_location
      label: Dag Processor Manager Log Location
      value: $MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/logs/dag_processor_manager/dag_processor_manager.log
      env: AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION
      description: |
        Where to send dag parser logs.
    - name: scheduler.child_process_log_directory
      label: Child Process Log Directory
      value: $MELTANO_PROJECT_ROOT/.meltano/utilities/airflow/logs/scheduler
      env: AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY
      description: |
        Where to send the logs of each scheduler process.
    - name: extension.airflow_home
      label: Airflow Home
      value: $MELTANO_PROJECT_ROOT/orchestrate/airflow
      env: AIRFLOW_HOME
      description: |
        The directory where Airflow will store its configuration, logs, and other files.
    - name: extension.airflow_config
      label: Airflow Home
      value: $MELTANO_PROJECT_ROOT/orchestrate/airflow/airflow.cfg
      env: AIRFLOW_CONFIG
      description: |
        The path where the Airflow configuration file will be stored.

Installation

# Install the extension
meltano install utility airflow

# explicitly seed the database, create default airflow.cfg, deploy the meltano dag orchestrator
meltano invoke airflow:initialize

# verify that airflow can be called via the extensions invoker
meltano invoke airflow version
# see what other commands are available
meltano invoke airflow --help

# create a airflow user with admin privs
meltano invoke airflow users create -u admin@localhost -p password --role Admin -e admin@localhost -f admin -l admin

# start the scheduler, backgrounding the process
meltano invoke airflow scheduler &
# start the triggerer, backgrounding the process
meltano invoke airflow triggerer &
# start the dag processor, backgrounding the process
meltano invoke airflow dag-processor &
# start the webserver, keeping it in the foreground
meltano invoke airflow api-server