CloudFormation Task definition is not updated during deployment

[rantoniuk]

Let me start by saying, that I'm not sure whether this report should go in this repository, CDK (for EcsService to provide a preserveTaskDefinitionRevision flag) or via AWS Support to CloudFormation team - or maybe I'm missing something.

I have a perfectly working setup that is using exactly as described in the README:

    - name: Fill in the new image ID in the Amazon ECS task definition
      id: task-def
      uses: aws-actions/amazon-ecs-render-task-definition@v1
      with:
        task-definition: task-definition.json
        container-name: my-container
        image: ${{ steps.build-image.outputs.image }}

    - name: Deploy Amazon ECS task definition
      uses: aws-actions/amazon-ecs-deploy-task-definition@v2
      with:
        task-definition: ${{ steps.task-def.outputs.task-definition }}
        service: my-service
        cluster: my-cluster

and an ECS service defined for simplicity like this:

    const taskDefinition = new ecs.Ec2TaskDefinition(this, 'TaskDef', {
      family: 'task-def',
      taskRole: taskRole,
      networkMode: ecs.NetworkMode.AWS_VPC,
    });

    const realtimeWsService = new ecs.Ec2Service(this, 'EcsService', {
      serviceName: 'Service',
      cluster: props.cluster,
      taskDefinition,
    });

Now imagine the following scenario - which is day to day work I believe, so I'm sure someone must have already had that problem, though I couldn't find it.

1. Deploy the CDK that creates the ECS service. This will in turn create the task-def:1 revision. Assuming all is fine with the service configuration, the service goes into steady state and CDK finishes with OK status - all good.
2. Now, start updating that service with this action, where the action will be updating the tag of the container image. Every run of GHA amazon-ecs-deploy-task-definition action, will create a new task-def
   revision, with the updated container image.

The service deployed, let's say with task-def:10, action finished, all good, right.

Now we get to the problem, Imagine that you need to change something in the Ec2Service definition, so the CDK - standard case, you need to increase the allocated memory or CPU.

If only the CPU allocation is changed and you run cdk diff, it will only show, as expected, that the service will be updated by only updating the CPU property.

However, when you actually deploy this, you'll see, that the service is restarted again with task-def:1 revision, instead of the last revision it was running with.

This is confirmed, if I go to CloudFormation console, look up the Stack and in there I can see in the stack resources, it is "attached" to the task-def:1 revision, instead of :10.

What am I missing? How to de-couple those two things, so that an unrelated CDK change doesn't revert the task-definition that is used by the service?

[amazreech]

Hi @rantoniuk, Thank you for reaching out with your question. For best assistance on this issue, I recommend contacting AWS Support as they should have the required expertise to address this. You can reach AWS Support through here: https://docs.aws.amazon.com/awssupport/latest/user/getting-started.html

Please do let me know if there is anything else I can help you with.

[rantoniuk]

I have submitted the Support Case, but I still would like someone from here to respond what is the usual way this action should be used together with a CDK-based container definition. I think the question I'm asking is a common problem so either I'm missing something or just need confirmation this is just how it is.

The workaround I currently implemented is to use non-immutable tags in the ECR repository. This way, the container is defined as: image:prod where the prod tag is continuously moving during publishing new revisions of the container.

This is acceptable, but it's not possible to use when using immutable tags.

The root cause of all of this is the fact, that a simple change like memory constraint update triggers a resource replacement instead of just updating the memory value:

 └─ [~] ContainerDefinitions (requires replacement)
     └─ @@ -40,7 +40,7 @@
        [ ]         }
        [ ]       }
        [ ]     },
        [-]     "Memory": 1024,
        [+]     "Memory": 2048,
        [ ]   }
        [ ] ]

[karanbokil]

Hi @rantoniuk ,

Not an official thread by any means, but would recommend you look at https://stackoverflow.com/questions/60049851/cdk-diff-does-not-diff-from-console-changes

cdk diff only detects changes between your local changes and the CFN stack that was deployed. It's not detecting changes between your local changes and what is actually the state of the resource server-side so to speak.

CDK/CFN assumes that any and all changes to a resource are done via CDK/CFN - anything done with a different tool is out-of-band and will cause drift

Using the amazon-ecs-deploy-task-definition is in fact an out-of-band change, so to reconcile your drift from cdk/cfn, you would need to follow one of the options mentioned here https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-resolve-drift.html