Terraform Provider Airflow

==========================

The Airflow provider is used to interact with the Airflow. The provider needs to be configured with the proper credentials before it can be used.

Example Usage

provider "airflow" {
  base_endpoint = "airflow.net"
  oauth2_token  = "token"
}

resource "airflow_variable" "default" {
  key   = "foo"
  value = "bar"
}

Authentication

Google Composer 1 Example (OAUTH2 identity token)

data "http" "client_id" {
  url = "composer-url"
}

resource "google_service_account" "example" {
  account_id = "example"
}

data "google_service_account_access_token" "impersonated" {
  target_service_account = google_service_account.example.email
  delegates              = []
  scopes                 = ["userinfo-email", "cloud-platform"]
  lifetime               = "300s"
}

provider "google" {
  alias        = "impersonated"
  access_token = data.google_service_account_access_token.impersonated.access_token
}

data "google_service_account_id_token" "oidc" {
  provider               = google.impersonated
  target_service_account = google_service_account.example.email
  delegates              = []
  include_email          = true
  target_audience        = regex("[A-Za-z0-9-]*\\.apps\\.googleusercontent\\.com", data.http.client_id.response_body)
}

provider "airflow" {
  base_endpoint = data.http.client_id.url
  oauth2_token  = data.google_service_account_id_token.oidc.id_token
}

Google Composer 2 Example (OAUTH2 access token)

Composer 2 changes how the API is accessed by the provider, you can just use the composer airflow web UI endpoint, and you can use a standard access token.

With service account impersionation

resource "google_service_account" "example" {
  account_id = "example"
}

data "google_service_account_access_token" "impersonated" {
  target_service_account = google_service_account.example.email
  delegates              = []
  scopes                 = ["userinfo-email", "cloud-platform"]
  lifetime               = "300s"
}

provider "google" {
  alias        = "impersonated"
  access_token = data.google_service_account_access_token.impersonated.access_token
}

data "google_client_config" "airflow" {
  provider = google.impersonated
}

provider "airflow" {
  base_endpoint = composer-url
  oauth2_token  = data.google_client_config.airflow.access_token
}

Using the default provider & service account

data "google_client_config" "airflow" {
  provider = google
}

provider "airflow" {
  base_endpoint = composer-url
  oauth2_token  = data.google_client_config.airflow.access_token
}

AWS MWAA (Managed Workflows for Apache Airflow) Example

For AWS MWAA environments, you need to use OAuth2 token authentication. The token can be generated using the AWS CLI:

aws mwaa create-web-login-token \
  --name YOUR_MWAA_ENVIRONMENT_NAME \
  --region us-east-1 \
  --query 'WebToken' \
  --output text

Then configure the provider:

data "external" "mwaa_token" {
  program = ["bash", "-c", <<-EOT
    aws mwaa create-web-login-token \
      --name my-mwaa-environment \
      --region us-east-1 \
      --query 'WebToken' \
      --output text | jq -R '{token: .}'
  EOT
  ]
}

provider "airflow" {
  base_endpoint = "https://YOUR-ENVIRONMENT-ID.c65.airflow.REGION.on.aws"
  oauth2_token  = data.external.mwaa_token.result.token
}

For more details on MWAA authentication, see the AWS documentation.

Airflow V3 (API v2)

In Airflow v3 (API v2) you cannot use basic auth directly anymore, you have to use OAUTH2 identity token. it can be generated via a user/password for a temporary jwt via:

curl -X POST https://airflow-server.net/auth/token \
  -H "Content-Type: application/json" \
  -d '{
    "username": "user",
    "password": "password"
  }'

Argument Reference

• base_endpoint - (Required) The Airflow API endpoint.
• oauth2_token - (Optional) An OAUTH2 identity token used to authenticate against an Airflow server. Conflicts with username and password
• username - (Optional) The username to use for API basic authentication. Conflicts with oauth2_token
• password - (Optional) The password to use for API basic authentication. Conflicts with oauth2_token
• disable_ssl_verification - (Optional) Disable SSL verification. Default is false
• base_path - (Optional) Base path for the Airflow API. Default is /api/v1. pass /api/v2 for Airflow v3 (API v2).

Running Acceptence Tests

Setting Up Local Environment

• See Official docs and run docker-compose up spin up a local airflow cluster.
• export AIRFLOW_BASE_ENDPOINT=http://localhost:8080
• export AIRFLOW_API_PASSWORD=airflow
• export AIRFLOW_API_USERNAME=airflow

Running Tests

Run make testacc