(aws-codestarnotifications): Creating SNS Notification rule fails when service account does not exist in the account

[cawofeso]

Describe the bug

When setting up notifications for Codepipeline rule on a Codepipeline where SNS is used as a target,
a codestar notifications service linked role needs to be available in the account otherwise the notification rule will fail to be created.

If the service linked role is not present (usually the case if notifications on a pipeline is being set up for the first time on an account) the service linked role is created automatically by Cloudformation which usually takes a bit of time. However before the service linked role has been created, Cloudformation goes off and attempts to create the Notification rule which fails as the service linked role is not yet present.

When I check the CreateNotificationRule event in Cloudtrail, I see the following error message in the responseElement

"AWS CodeStar Notifications could not create the AWS CloudWatch Events managed rule in your AWS account. If this is your first time creating a notification rule, the service-linked role for AWS CodeStar Notifications might not yet exist. Creation of this role might take up to 15 minutes. Until it exists, notification rule creation will fail. Wait 15 minutes, and then try again. 
If this is is not the first time you are creating a notification rule, there might be a problem with a network connection, or one or more AWS services might be experiencing issues. 
Verify your network connection and check to see if there are any issues with AWS services in your AWS Region before trying again."

There has been previous reports of this issue where it was assumed that adding a dependency on the topic role
would fix the issue but the actual root of the problem is that the service linked role needs to be created first. (#29484)

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

Deployment is successful

Current Behavior

Deployment fails and get the following error in Cloudformation:
Invalid request provided: AWS::CodeStarNotifications::NotificationRule

Reproduction Steps

1. Ensure there is no codestar notifications service linked role in the aws account (if there is one delete it)
2. Attempt to deploy a pipeline with a notification rule where an SNS topic is set as the target e.g

const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
    pipelineName: 'MyPipeline',
    ...
});
const topic = new sns.Topic(this, 'Topic');
const rule = new NotificationRule(this, 'NotificationRule', {
    ...,
    targets: [topic]
});

3. Deployment should fail and when you check the CreateNotificationRule event in Cloudtrail you should see the error message in the responseElement that it failed to create the CloudWatch Events managed rule

Possible Solution

A possible way round this is to have a custom resource that goes off and creates the service linked role and
add a dependency on the notification rule so that the service linked role is created before creating the notification rule.

However, it would be nice if there is a much simpler solution to this issue and open to suggestions. (is it possible to add a dependency on a service link role without using a custom resource?)

(I'm also happy to create attempt to PR to fix this issue once a solution is agreed upon)

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.189.0

AWS CDK CLI version

2.1007.0

Node.js Version

20

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[ykethan]

Hey @cawofeso, thank you for reaching out and providing us this information. I was able to reproduce this behavior using the following

Reproduction:

import * as cdk from 'aws-cdk-lib';
import * as codepipeline from 'aws-cdk-lib/aws-codepipeline';
import * as codepipeline_actions from 'aws-cdk-lib/aws-codepipeline-actions';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as notifications from 'aws-cdk-lib/aws-codestarnotifications';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as codebuild from 'aws-cdk-lib/aws-codebuild';
import { Construct } from 'constructs';

export class NotificationTestStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create source bucket for the pipeline
    const sourceBucket = new s3.Bucket(this, 'SourceBucket', {
      versioned: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });

    // Create artifact for the pipeline
    const sourceOutput = new codepipeline.Artifact('SourceArtifact');

    // Create a simple CodeBuild project
    const buildProject = new codebuild.PipelineProject(this, 'BuildProject', {
      environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_5_0,
      },
    });

    // Create a pipeline with required stages and actions
    const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
      pipelineName: 'TestPipeline',
      stages: [
        {
          stageName: 'Source',
          actions: [
            new codepipeline_actions.S3SourceAction({
              actionName: 'S3Source',
              bucket: sourceBucket,
              bucketKey: 'source.zip',
              output: sourceOutput,
            }),
          ],
        },
        {
          stageName: 'Build',
          actions: [
            new codepipeline_actions.CodeBuildAction({
              actionName: 'BuildAction',
              project: buildProject,
              input: sourceOutput,
            }),
          ],
        },
      ],
    });

    // Create an SNS topic to use as a notification target
    const topic = new sns.Topic(this, 'Topic', {
      topicName: 'notification-test-topic',
    });

    // Create a notification rule
    const notificationRule = new notifications.NotificationRule(
      this,
      'NotificationRule',
      {
        notificationRuleName: 'TestNotificationRule',
        detailType: notifications.DetailType.FULL,
        events: ['codepipeline-pipeline-pipeline-execution-failed'],
        source: pipeline,
        targets: [topic],
      },
    );

    // Output the notification rule ARN for reference
    new cdk.CfnOutput(this, 'NotificationRuleArn', {
      value: notificationRule.notificationRuleArn,
    });
  }
}

CloudTrail event:

"responseElements": {
"Message": "AWS CodeStar Notifications could not create the AWS CloudWatch Events managed rule in your AWS account. If this is your first time creating a notification rule, the service-linked role for AWS CodeStar Notifications might not yet exist. Creation of this role might take up to 15 minutes. Until it exists, notification rule creation will fail. Wait 15 minutes, and then try again. If this is is not the first time you are creating a notification rule, there might be a problem with a network connection, or one or more AWS services might be experiencing issues. Verify your network connection and check to see if there are any issues with AWS services in your AWS Region before trying again."
},

On analyzing the Cloudtrail and CloudFormation events, its does appear:

1. CloudFormation tries to create the notification rule
2. The CodeStar Notifications service attempts to create its service-linked role automatically
3. However, CloudFormation doesn't know to wait for this asynchronous role creation
4. The notification rule creation fails because the role isn't ready in time

To confirm the above behaviour on further investigation

1. When deploying a NotificationRule with improper template for example:

{
  "Resources": {
    "NotificationRule": {
      "Type": "AWS::CodeStarNotifications::NotificationRule",
      "Properties": {
        "EventTypeIds": [
          "codepipeline-pipeline-pipeline-execution-failed"
        ],
        "DetailType": "FULL",
        "Resource": "arn:aws:codebuild:us-east-1::project/test",
        "Targets": [
          {
            "TargetType": "SNS",
            "TargetAddress": "arn:aws:sns:us-east-1::test"
          }
        ],
        "Name": "TestNotificationRule"
      }
    }
  }
}

here i am providing incorrect event type id, CloudFormation always throws

Resource handler returned message: "Invalid request provided: AWS::CodeStarNotifications::NotificationRule" (RequestToken: aa196ff8-e7e9-b65b-228d-eabb12aeaf02, HandlerErrorCode: InvalidRequest)

on diving into cloudtrail throws

The following event type could not be added to the notification rule: codepipeline-pipeline-pipeline-execution-failed. That event type is not valid for the following resource: arn:aws:codebuild:us-east-1::project/test.

2. similarly, CloudFormation immediately throws a similar error when the service is currently create a service role without waiting role to finish creating.
   example template:

{
  "Resources": {
    "NotificationRule": {
      "Type": "AWS::CodeStarNotifications::NotificationRule",
      "Properties": {
        "EventTypeIds": [
          "codebuild-project-build-state-failed"
        ],
        "DetailType": "FULL",
        "Resource": "arn:aws:codebuild:us-east-1::project/test",
        "Targets": [
          {
            "TargetType": "SNS",
            "TargetAddress": "arn:aws:sns:us-east-1::test"
          }
        ],
        "Name": "TestNotificationRule"
      }
    }
  }
}

AWS CodeStar Notifications could not create the AWS CloudWatch Events managed rule in your AWS account. If this is your first time creating a notification rule, the service-linked role for AWS CodeStar Notifications might not yet exist. Creation of this role might take up to 15 minutes. Until it exists, notification rule creation will fail. Wait 15 minutes, and then try again. If this is is not the first time you are creating a notification rule, there might be a problem with a network connection, or one or more AWS services might be experiencing issues. Verify your network connection and check to see if there are any issues with AWS services in your AWS Region before trying again.

Opened a ticket with CloudFormation providing information on these behaviours: V1773552096

In the meantime, we'll discuss this internally with the CDK team to determine the best course of actions such as utilizing custom resources