L2 CDK Construct for AirFlow

[bodazyphir]

Describe the feature

There does not seem to be an L2 construct for AirFlow in the CDK library "aws-cdk-lib". The only way to code an airflow environment is with the CloudFormation L1 construct "CfnEnvironment". This however does not facilitate dependency chaining as you cannot make an L1 construct dependant on L2 constructs , please can this be remedied.

Use Case

Creating L2 AirFlow infrastructure and chaining it to other L2 infra as a dependancy.

Proposed Solution

Write an L2 method for AirFlow and add it to the node aws-cdk-lib library.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.157.0

Environment details (OS name and version, etc.)

Ubuntu 22.04.3 LTS

[pahud]

Hi

Are you proposing the L2 for MWAA?
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-mwaa-environment.html
https://github.com/aws/aws-cdk/tree/main/packages/aws-cdk-lib/aws-mwaa

[bodazyphir]

Hi Pahud,

Yes that is bang on the money.

Regards,
Stephen

[Jacco]

I would like to contribute to this L2 construct.

Would that go through CDK RFCs first? Below a "working backward" readme :-)

Apart from that following features are also interesting:

• Create BucketDeployment for a dags folder (also startup script etc)
• maybe: secretsmanager / parameterstore backend configuration
• maybe: plugins constructor?
• maybe: when EndpointManagement is CUSTOMER, create the necessary resources and hand back to adjust

AWS MWAA Construct Library

This construct library allows you to define AWS MWAA environments.

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow'
});

Airflow version and environment class

If not specified the MWAA environment will use the latest supported Airflow version and
a class of mw1.small. To obtain a MWAA environment with an older Airflow version or
of a different size:

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow',
  airflowVersion: AirflowVersion.VER_2_10_1,
  environmentClass: EnvironmentClass.MW1_LARGE,
});

It is also possible to use an Apache Airflow version that is not yet supported by the version of aws-cdk-lib being used. Then you can use the implemented escape hatch:

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow',
  airflowVersion: AirflowVersion.of('5.8.9'),
});

Such an escape hatch also exists for the environment class:

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow',
  environmentClass: EnvironmentClass.of('mw2.4xlarge'),
});

The Airflow version also determines the python version being used by Airflow. Each predefined AirflowVersion
has a pythonVersion field and also supports retrieving the link to the constrainsts file for the python
dependencies:

const pyver: string = AirflowVersion.VER_2_10_1.pythonVersion;
const link: string = AirflowVersion.VER_2_10_1.constraintsFileLink;

?? JPK environmentClass determines some other properties: default concurrent tasks, DAG capacity, worker/scheduler/webserver/database vcpu&mem

Execution Role

MWAA environments assume an IAM role when accessing AWS resources. In CDK by default, MWAA environments will use an autogenerated Role if one is not provided.

The autogenerated Role is automatically given permissions necessary for the MWAA environment to perform necessary actions:

• allow to read from source bucket
• deny to list all buckets
• allow read/write to log groups associated with this environment
• allow describing loggroups
• allow read/write to sqs queues starting with airflow-celery- (in any AWS account)
• allow key usage of KMS keys NOT in the current AWS account via the sqs service
• allow publishing cloudwatch metrics

To reference the autogenerated Role:

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow'
});

const role = env.role; // the Role

To add iam statements to the role you can use

    env.addToRolePolicy(...)

You can also provide your own IAM role. Provided IAM roles will not automatically
be given any permissions. To provide a role and grant it appropriate permissions:

const myRole = new iam.Role(this, 'MyRole', {
  assumedBy: new iam.CompositePrincipal(
    new iam.ServicePrincipal('airflow.amazonaws.com'),
    new iam.ServicePrincipal('airflow-env.amazonaws.com'),
  )
});

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow',
  executionRole: myRole
});

myRole.addToPrincipalPolicy(new iam.PolicyStatement({
    actions: ['airflow:PublishMetrics'],
    resources: [this.environmentArn]
}))

Source bucket

MWAA environments are associated with an S3 Bucket where Airflow assets are stored. In CDK by default, MWAA environments will use an autogenerated S3 Bucket if one is not provided.

The autogenerated Bucket will have versioning enabled. Its resource policy allows the execution policy to read from it.

To reference the autogenerated S3 Bucket:

const lambdaRole = new iam.Role(this, 'LambdaRole', {
  assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com')
});

const env = new mwaa.Environment(this, 'MyEnvironment', {
  environmentName: 'airflow',
});

env.grantReadWrite(lambdaRole, 'dags/*')

VPC and SecurityGroup

todo

Folders in the source bucket

todo

Logging configuration

todo

Airflow configuration

todo

Maintenance window

todo

Endpoint management

todo