cloudwatch: subscription filter creation is concurrent to CanInvokeLambda permission creation

[Lanayx]

Describe the bug

When we add subscription filter to the cloudwatch group log there are two operations: add permission to invoke lambda from cloudwatch service (CanInvokeLambda) and add subscription filter itself. The order is correct - add permission starts first and then create subscription filter. The problem is that adding permission can still be in progress and filter creation fails with the error "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"

Expected Behavior

I expect deploy to wait permission creation, so filter creation will never fail.

Current Behavior

Sometimes deploy succeeds, sometimes it fails

Reproduction Steps

logGroup.AddSubscriptionFilter("myid", SubscriptionFilterOptions
            (
                FilterPattern = FilterPattern.Any(
                    FilterPattern.StringValue("$.log_severity", "=", "ERROR"),
                    FilterPattern.StringValue("$.log_severity", "=", "WARN"),
                    FilterPattern.StringValue("$.log_severity", "=", "FATAL")
                ),
                Destination = LambdaDestination(notificationLambda)
            ))

Possible Solution

Wait for permission creation

Additional Information/Context

Amazon.CDK.Lib 2.44.0

CDK CLI Version

2.50.0 (build 4c11af6)

Framework Version

No response

Node.js Version

18.12.1

OS

Linux

Language

.NET

Language Version

No response

Other information

failed log example:

30-Nov-2022 13:01:10	ec-m-tech-dev-common-MainStack | 16/80 | 1:01:07 PM | CREATE_IN_PROGRESS   | AWS::Lambda::Permission                         | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) 
30-Nov-2022 13:01:10	ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS   | AWS::Lambda::Permission                         | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation Initiated
30-Nov-2022 13:01:10	ec-m-tech-dev-common-MainStack | 16/80 | 1:01:08 PM | CREATE_IN_PROGRESS   | AWS::Logs::SubscriptionFilter                   | MainStack/DriveWealthEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource creation Initiated
30-Nov-2022 13:01:10	ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED        | AWS::Logs::SubscriptionFilter                   | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterId0D1F460D) Resource handler returned message: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function. (Service: CloudWatchLogs, Status Code: 400, Request ID: 3520c986-93b0-4c3e-a6d4-266bfad60a37)" (RequestToken: a31769e1-5aec-d85c-30ba-4c81e0eca8c1, HandlerErrorCode: InternalFailure)
30-Nov-2022 13:01:10	ec-m-tech-dev-common-MainStack | 16/80 | 1:01:09 PM | CREATE_FAILED        | AWS::Lambda::Permission                         | MainStack/DWEventListenerId/DWEventListenerId-LogGroup/NotificationSubscriptionFilterId/CanInvokeLambda (DWEventListenerIdDWEventListenerIdLogGroupNotificationSubscriptionFilterIdCanInvokeLambda24FE7AF1) Resource creation cancelled

[peterwoodworth]

Thanks for the bug report. Are you able to work around this issue by making the Permission depend on the SubscriptionFilter?

You should be able to access these methods and resources you need to add dependencies on with escape hatches. Here's an example in TypeScript which I have verified adds the dependency:

    // Given new log group + filter...
    const log = new LogGroup(this, 'lg')
    const filter = log.addSubscriptionFilter('filter', {
      destination: new LambdaDestination(new MyFunction(this, 'fn').fn),
      filterPattern: FilterPattern.any(FilterPattern.stringValue("$.log_severity", "=", "ERROR"))
    });

    // Add dependency. The permission is child of the filter
    (filter.node.findChild('CanInvokeLambda') as CfnPermission).addDependsOn(filter.node.defaultChild as CfnSubscriptionFilter);

In CDK code the subscription filter is created here, and it also binds the destination:

aws-cdk/packages/@aws-cdk/aws-logs/lib/subscription-filter.ts

Line 54 in 7690f43

export class SubscriptionFilter extends Resource {

aws-cdk/packages/@aws-cdk/aws-logs-destinations/lib/lambda.ts

Line 24 in 7690f43

public bind(scope: Construct, logGroup: logs.ILogGroup): logs.LogSubscriptionDestinationConfig {

We could add the dependency somewhere here in CDK code to prevent anyone from having to do this manually

[Lanayx]

Thank you, for the workaround suggested, I'll try it, just wanted to clarify - why should Permission depend on the SubscriptionFilter? It seems that it should be vise versa - SubscriptionFilter to depend on Permission (since permission should be created first)

[peterwoodworth]

Sorry, I mixed that up you're right. Try the other way around

[Yashodhan-Asu]

@peterwoodworth, @Lanayx other way around works.
(filter.node.defaultChild as CfnSubscriptionFilter).addDependsOn(filter.node.findChild('CanInvokeLambda') as CfnPermission);

[alexandervandekleutab]

@Yashodhan-Asu your solution also worked for me. Would love to see this added to at least the docs, if not the source code.

[PetarRAutoAddress]

Above workaround in C#

{ CfnSubscriptionFilter cfnSubscriptionFilter = (logsSubscriptionFilter.Node.DefaultChild as CfnSubscriptionFilter) ?? throw new InvalidOperationException(); CfnPermission cfnPermission = (logsSubscriptionFilter.Node.FindChild("CanInvokeLambda") as CfnPermission) ?? throw new InvalidOperationException(); cfnSubscriptionFilter.AddDependency(cfnPermission); }

[moltar]

If someone is a real stickler for type safety: 😁

assert(filter.node.defaultChild instanceof CfnSubscriptionFilter, 'Not a SubscriptionFilter.')
const canInvokeLambda = filter.node.findChild('CanInvokeLambda')
assert(canInvokeLambda instanceof CfnPermission, 'Cannot find CanInvokeLambda.')
filter.node.defaultChild.addDependency(canInvokeLambda)

[nikhil-zadoo]

The issue has been fixed in the newer CDK versions. Don't know exactly in which version the fix was pushed. Although I can confirm version 2.83.1 has the permission dependency added out of box.