aws-ecs: Resource handler returned message: "Invalid request provided: The target group with targetGroupArn <redacted> does not have an associated load balancer."

[BwL1289]

Describe the bug

With this workaround for redirecting HTTP to HTTPS on an ALB, I indeterminately experience the following:

CoreStackDev | 28/70 | 5:02:27 PM | UPDATE_FAILED        | AWS::ECS::Service                             | WebAppSvc/FlaskAppService/ControlPlane/ApplicationLoadBalancerFargateSvc/AlbFargateSvc/Service/Service (WebAppSvcFlaskAppServiceControlPlaneApplicationLoadBalancerFargateSvcAlbFargateSvcServiceADE54F9D) Resource handler returned message: "Invalid request provided: The target group with targetGroupArn <redacted> does not have an associated load balancer. (Service: Ecs, Status Code: 400, Request ID: 91fd6146-d2f3-4534-bbdb-893469be2a1d) (SDK Attempt Count: 1)" (RequestToken: 25bc3e51-6110-4b81-8256-f4c02c60a863, HandlerErrorCode: InvalidRequest)

I tested redeploying multiple times (and changing the ECS Service, ALB, Task definition logicalids to recreate the resources) to no avail.

It looks like there's either:

1. A race condition (regression)? somewhere. What's strange is this was working fine for a few days with multiple successful redeployments in between the original deployment and experiencing the error.
2. Some other reason why cloudformation thinks an old ALB target group exists when it does not

After removing the listener redirection (lb.ListenerAction.redirect) the deployment succeeded again.

My ticket is related.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Successful deployment

Current Behavior

Indeterminate Invalid request provided: The target group with targetGroupArn <arn> does not have an associated load balancer.

Reproduction Steps

1. Deploy an ALB with ecs_patterns.ApplicationLoadBalancedFargateService with protocol=lb.ApplicationProtocol.HTTP,
2. Use this workaround to redirect HTTP to HTTPSand deploy
3. Deploy a few times...
4. Maybe experience error?

Possible Solution

No response

Additional Information/Context

"userAgent": "cloudformation.amazonaws.com",
    "errorCode": "InvalidParameterException",
    "errorMessage": "The target group with <redacted>/CoreSt-WebAp-OBAY5UVSDTFW/5f79f6f39969fa92 does not have an associated load balancer.",
    "requestParameters": {
        "capacityProviderStrategy": [
            {
                "capacityProvider": "FARGATE",
                "weight": 1,
                "base": 1
            }
        ],
        "cluster": "CoreStackDev-WebAppSvcFlaskAppServiceControlPlaneEcsClusterSvcEcsClusterEcsClusterSvcB291F355-xtUAqFpdBbkj",
        "deploymentConfiguration": {
            "deploymentCircuitBreaker": {
                "enable": true,
                "rollback": true
            },
            "maximumPercent": 200,
            "minimumHealthyPercent": 100,
            "alarms": {
                "rollback": false,
                "enable": false
            }
        },
        "desiredCount": 1,
        "enableECSManagedTags": true,
        "enableExecuteCommand": true,
        "availabilityZoneRebalancing": "DISABLED",
        "forceNewDeployment": false,
        "healthCheckGracePeriodSeconds": 60,
        "loadBalancers": [
            {
                "targetGroupArn": "<redacted>/CoreSt-WebAp-OBAY5UVSDTFW/5f79f6f39969fa92",
                "containerName": "EcsTaskDefContainerSvcEcsContainerDef",
                "containerPort": 80
            },
            {
                "targetGroupArn": "<redacted>/CoreSt-WebAp-XB77YZBVGDH8/559a5660e756f528",
                "containerName": "EcsTaskDefContainerSvcEcsContainerDef",
                "containerPort": 80
            }
        ],

CDK CLI Version

2.180.0

Framework Version

No response

Node.js Version

v22.12.0

OS

Mac

Language

Python

Language Version

No response

Other information

No response

[ykethan]

Hey @BwL1289, thank you for reaching out.

After examining the AWS CDK codebase, the following could be cause of the intermittent The target group with targetGroupArn does not have an associated load balancer error when using HTTP to HTTPS redirection.

Code Analysis:

1. ECS Service Dependency on Target Group
   In packages/aws-cdk-lib/aws-ecs/lib/base/base-service.ts, the ECS service creates a dependency on the target group's load balancer attachment:

private attachToELBv2(targetGroup: elbv2.ITargetGroup, containerName: string, containerPort: number): elbv2.LoadBalancerTargetProps {
  // ...
  this.loadBalancers.push({
    targetGroupArn: targetGroup.targetGroupArn,
    containerName,
    containerPort,
  });

  // Service creation can only happen after the load balancer has
  // been associated with our target group(s), so add ordering dependency.
  this.resource.node.addDependency(targetGroup.loadBalancerAttached);
  
  // ...
}

This line this.resource.node.addDependency(targetGroup.loadBalancerAttached) creates the dependency that should ensure the target group is associated with a load balancer before the service is created.

2. Target Group Load Balancer Attachment Tracking
   In packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/shared/base-target-group.ts, the load balancer attachment dependencies are managed:

/**
 * Configurable dependable with all resources that lead to load balancer attachment
 */
protected readonly loadBalancerAttachedDependencies = new DependencyGroup();

/**
 * List of constructs that need to be depended on to ensure the TargetGroup is associated to a load balancer
 */
public get loadBalancerAttached(): IDependable {
  return this.loadBalancerAttachedDependencies;
}

3. Listener Registration in Target Group
   In packages/aws-cdk-lib/aws-elasticloadbalancingv2/lib/alb/application-target-group.ts, when a listener is added:

/**
 * Register a listener that is load balancing to this target group.
 *
 * Don't call this directly. It will be called by listeners.
 */
public registerListener(listener: IApplicationListener, associatingConstruct?: IConstruct): void {
  // ...
  this.listeners.push(listener);
  this.loadBalancerAttachedDependencies.add(associatingConstruct ?? listener);
}

The issue likely occurs because:

When using the HTTP to HTTPS redirection workaround, you're modifying or replacing the action on an existing listener after it's been created and registered with the target group.

You create a service with ApplicationLoadBalancedFargateService that sets up HTTP listener
You then modify the HTTP listener to redirect to HTTPS
You add an HTTPS listener with a forward action to the same target group
The following shows two target groups being registered with the ECS service:

"loadBalancers": [
  { "targetGroupArn": ".../CoreSt-WebAp-OBAY5UVSDTFW/5f79f6f39969fa92" },
  { "targetGroupArn": ".../CoreSt-WebAp-XB77YZBVGDH8/559a5660e756f528" }
]

This suggests that when modifying listeners and actions, the dependency tracking for target group associations may be causing the issue.

Tried reproducing the issue using the code provided in #5583 (comment) but was not able to observe this error message.

Will bring this up internally for any additional feedback but do provide us any additional information that could help us dive into this issue.

[BwL1289]

@ykethan I agree with your analysis. I get a warning during deployment / synth about the listener modification.

As indicated by the other GH issues I've created on the topic, CDK doesn't natively support this use case without the workaround above, so I'm not sure how to mitigate the error.

For context, I need this to comply with the SOC2 security standard, so appreciate support from AWS to either implement native support or find another workaround to this current workaround.

[ykethan]

@BwL1289 thank you patience while we dive into this issue. I was successfully able to reproduce the "The target group with targetGroupArn does not have an associated load balancer" error. Here's a detailed reproduction and explanation of the issue:

Reproduction Code:

// Create HTTP listener with target group first
const httpListener = loadBalancer.addListener('HttpListener', {
  port: 80,
  open: true,
  defaultTargetGroups: [targetGroup], // Associate target group with listener
});

// Create HTTPS listener with the same target group
const httpsListener = loadBalancer.addListener('HttpsListener', {
  port: 443,
  certificates: [certificate],
  open: true,
  defaultTargetGroups: [targetGroup],
});

// Create Fargate service
const fargateService = new ecs.FargateService(this, 'Service', {
  cluster,
  taskDefinition,
  desiredCount: 1,
  assignPublicIp: false,
});

// Register service with target group
targetGroup.addTarget(fargateService);

// PROBLEM TRIGGER: Modifying HTTP listener to redirect AFTER the target group 
// has been associated and AFTER the service has been registered
httpListener.addAction('HttpRedirect', {
  action: lb.ListenerAction.redirect({
    port: '443',
    protocol: 'HTTPS',
    permanent: true,
  }),
});

During deployment, we get this warning:

[Warning at /MyProjectStack/ALB/HttpListener] A default Action already existed on this Listener and was replaced.`

And we get the error:

Resource handler returned message: "Invalid request provided: The target group with targetGroupArn does not have an associated load balancer. (Service: Ecs, Status Code: 400)"

Console testing:

1. Create a ALB with 2 listeners HTTP and HTTPS. Added them to the same target group.
2. Once ALB is created. Select the HTTP:80 listener(checkbox)
3. click, Manage Listener -> Edit listener

4. save changes.

No errors observed on console

Root Cause
The issue occurs due to the sequence of operations:

1. We create a target group and associate it with an HTTP listener
2. When we later modify the HTTP listener to redirect to HTTPS, we're replacing its default action
3. This disrupts the dependency tracking between the target group and load balancer in CDK
4. When deploying, CloudFormation's ordering of operations can sometimes cause the ECS service to be created/updated before the target group is fully associated with a load balancer

The warning about replacing a default action does appear to be the key indicator for this issue. But similar to #34235 modifying listeners after creation, appears to be causing this error.
Marking as P1 Bug

[BwL1289]

@ykethan thank you.

[BwL1289]

Is there an update on this? We are blocked from completing SOC-2 compliance.