Aws cdk does not exactly deploy what's on diff.

[IhebMarnaoui]

Describe the bug

I have noticed lately that every time I deploy a change that I am doing, a record on Route53 is disappearing even though it shouldn't. I deployed this on 3 different environments (and different projects) and I have had the same behaviour each time. the operation is basically changing the location of this TXTRecord location in my IAC from a stack to a construct that is being called within the stack itself without changing anything else in the TXTRecord code itself. It is basically an operation of cut-paste.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The diff shows that this record (shown in the screenshot) should be destroyed from the initial location and gets created in another one.

Current Behavior

What actually happens in that the resource (TXTRecord) gets deleted but never recreated again.

Reproduction Steps

Move the record code from a stack to a construct.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.150.0 (build 3f93027)

Framework Version

No response

Node.js Version

v20.9.0

OS

macOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi

I will try to reproduce this in my account but any chance you could provide minimal code snippets that I can simply paste into my IDE and reproduce that? It would make sure we are on the same track with the same code snippets addressing this issue. Thank you.

[pahud]

This is my testing code.

Initial deploy:

export class MyStack9 extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const hostedZone = new route53.HostedZone(this, 'HostedZone', {
      zoneName: 'exampleee.com',
    });

    // This TXTRecord will be moved to a nested construct
    const txtRecord = new route53.TxtRecord(this, 'TXTRecord', {
      zone: hostedZone,
      recordName: 'my-txt-record',
      values: ['value1', 'value2'],
    });
  }
}

Now, create a construct for the txtRecord

export interface DemoConstructProps {
  hostedZone: route53.HostedZone;
}

export class DemoConstruct extends Construct {
  constructor(scope: Construct, id: string, props: DemoConstructProps) {
    super(scope, id);

    // This TXTRecord will be moved to a nested construct
    new route53.TxtRecord(this, 'TXTRecord', {
      zone: props.hostedZone,
      recordName: 'my-txt-record',
      values: ['value1', 'value2'],
    });    

  }
}


export class MyStack9 extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const hostedZone = new route53.HostedZone(this, 'HostedZone', {
      zoneName: 'exampleee.com',
    });

    new DemoConstruct(this, 'DemoConstruct', {
      hostedZone,
    })

    // // This TXTRecord will be moved to a nested construct
    // const txtRecord = new route53.TxtRecord(this, 'TXTRecord', {
    //   zone: hostedZone,
    //   recordName: 'my-txt-record',
    //   values: ['value1', 'value2'],
    // });
  }
}

npx cdk diff

Resources
[-] AWS::Route53::RecordSet TXTRecord TXTRecordA3427B34 destroy
[+] AWS::Route53::RecordSet DemoConstruct/TXTRecord DemoConstructTXTRecord3FF42E13 

On cdk deploy

It

1:46:49 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack | dummy-stack2
1:46:52 PM | CREATE_IN_PROGRESS   | AWS::Route53::RecordSet  | DemoConstruct/TXTRecord

And yes, that txtRecord is deleted.

I feel this is a bug of cloudformation, looks like it would first create a new txtRecord with the same recordName, which should not be allowed but CFN does not return any error and moves on to delete the existing one. Eventually, no txtRecord would exist.

I'll share more details later.

[pahud]

OK what we know:

1. When you move a resource from stack scope to construct scope, what's happening in CDK is that it's logicalID would be changed and CDK would consider them as two different resources(one to be created and the other to be destroyed)
2. I noticed cloudformation was trying to create first but it didn't throw any error(It should have). I would expect it to throw the error as it is trying to create the same duplicate TxtRecord.
3. As the creation operation does not fail, the destroy operation would delete that resource.
4. Eventually, no TxtRecord would be left.

I also tried similar use case with sqs.Queue with an explicit queueName

firstly, create a sqs.Queue in a Stack

    const queue = new sqs.Queue(this, 'Queue', {
      queueName: 'my-queue',
    });

Then, define a construct for that and create that construct in the stack

export class DemoConstruct extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    new sqs.Queue(this, 'Queue', {
      queueName: 'my-queue',
    })
  }
}

export class MyStack9 extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new DemoConstruct(this, 'DemoConstruct');
  }
}

On cdk diff I see

Resources
[-] AWS::SQS::Queue Queue Queue4A7E3555 destroy
[+] AWS::SQS::Queue DemoConstruct/Queue DemoConstructQueue638BE825 

while on cdk deploy, CFN tried to create the resource first but failed because of duplicate resource

2:26:44 PM | CREATE_FAILED        | AWS::SQS::Queue    | DemoConstructQueue638BE825
Resource handler returned message: "Resource of type 'AWS::SQS::Queue' with identifier 'my-queue' already exists." (RequestToken: aa0c54ea-33e7-56c4-
811e-04ffd57dc9f5, HandlerErrorCode: AlreadyExists)

So I think it's a bug of CFN.

@IhebMarnaoui as you mentioned this is a regression of CDK. Do you have a specific CDK version number that was working with exactly the same code?

[pahud]

internal tracking -
V1573725376
V216719100

[IhebMarnaoui]

Hello @pahud, thank you for giving attention to this.
What you have mentioned in your first comment is exactly the same thing that happened to me. Almost same code was used.
This is the snippet from my code that would cause the bug mentioned:

    new route53.TxtRecord(this, `${projectName}-dmarc-text-record`, {
      zone: zone.hostedZone,
      recordName: `_dmarc.${domain}`,
      values: [`v=DMARC1; p=quarantine; pct=100; rua=mailto:${dmarcEmailAddress}`],
    });

@IhebMarnaoui as you mentioned this is a regression of CDK. Do you have a specific CDK version number that was working with exactly the same code?

No, sorry, I don't remember running into the same case before. This could never have worked before.

[pahud]

@IhebMarnaoui OK I got it.

We are actively tracking this issue internally as this could be a CFN bug. From my perspective, I would expect CFN would fail just like the sqs Queue sample.

I'll circle back if I have new updates.