ECS CapacityProvider update fails with `UPDATE_FAILED`

[MeganYTan]

Describe what happened

When updating an existing aws.ecs.CapacityProvider resource’s autoScalingGroupProvider.managedScaling.targetCapacity value via Pulumi, the update consistently fails.

Command executed:

• pulumi up

Change applied:

• Modify managedScaling.targetCapacity from 50 to 80 on an existing ECS Capacity Provider that is already attached to an ECS cluster using an Auto Scaling Group (ASG) capacity provider strategy.
• The Pulumi preview shows an in-place update of the aws.ecs.CapacityProvider (no replacement).

During pulumi up, the update fails with the following error surfaced from the AWS ECS API:

aws:ecs:CapacityProvider (<capacity-provider-name>):
    error: 1 error occurred:
        * updating urn:pulumi:<stack-name>::<project-name>::aws:ecs/capacityProvider:CapacityProvider::<capacity-provider-name>: 1 error occurred:
        * waiting for ECS Capacity Provider (arn:aws:ecs:<region>:<account-id>:capacity-provider/<capacity-provider-name>) update: unexpected state 'UPDATE_FAILED', wanted target 'UPDATE_COMPLETE'. last error: The capacity provider cannot be updated due to an internal error. Try again.

This behavior has been observed with both of the following provider versions:

• @pulumi/aws v6.81.0
• @pulumi/aws v7.10.0

The same configuration change can be successfully applied through the AWS Console using a multi-step manual sequence (see “Additional context”), which suggests a subtle interaction between ECS cluster capacity provider attachments and the Capacity Provider’s managed scaling settings that the provider may not currently be handling optimally.

Sample program

const capacityProvider = new aws.ecs.CapacityProvider(
    `example-asg-autoscaling-provider`,
    {
      name: `example-asg-autoscaling-provider`,
      autoScalingGroupProvider: {
        autoScalingGroupArn: exampleasg.arn,
        managedTerminationProtection: 'ENABLED',
        managedScaling: {
          maximumScalingStepSize: 1000,
          minimumScalingStepSize: 1,
          status: 'ENABLED',
          targetCapacity: 80,
        },
      },
    }

Log output

Diagnostics:
  aws:ecs:CapacityProvider (example-asg-capacity-provider):
    error: sdk-v2/provider2.go:566: sdk.helper_schema: waiting for ECS Capacity Provider (arn:aws:ecs:<region>:<account-id>:capacity-provider/example-asg-capacity-provider) update: unexpected state 'UPDATE_FAILED', wanted target 'UPDATE_COMPLETE'. last error: The capacity provider cannot be updated due to an internal error. Try again.

  aws:ecs:CapacityProvider (example-asg-capacity-provider):
    error: sdk-v2/provider2.go:572: sdk.helper_schema: waiting for ECS Capacity Provider (arn:aws:ecs:<region>:<account-id>:capacity-provider/example-asg-capacity-provider) update: unexpected state 'UPDATE_FAILED', wanted target 'UPDATE_COMPLETE'. last error: Cluster <example-ecs-cluster> is processing a previous update. Wait for the cluster attachments to be in UPDATE_COMPLETE or UPDATE_FAILED state and try again.
    error: 1 error occurred:
        * updating urn:pulumi:<stack-name>::<project-name>::aws:ecs/capacityProvider:CapacityProvider::example-asg-capacity-provider: 1 error occurred:
        * waiting for ECS Capacity Provider (arn:aws:ecs:<region>:<account-id>:capacity-provider/example-asg-capacity-provider) update: unexpected state 'UPDATE_FAILED', wanted target 'UPDATE_COMPLETE'. last error: The capacity provider cannot be updated due to an internal error. Try again.

This pattern repeats for each stack where the managedScaling.targetCapacity setting is modified on an in-use Capacity Provider.

Affected Resource(s)

• aws.ecs.CapacityProvider
• (Indirectly) aws.ecs.ClusterCapacityProviders

Output of pulumi about

Not provided

Additional context

Operational context and observations:

• The issue occurs across multiple stacks in the same AWS account and region when changing managedScaling.targetCapacity on an existing Capacity Provider that is already attached to an ECS cluster.
• The ECS API error indicates:
  • An internal error updating the capacity provider:
    The capacity provider cannot be updated due to an internal error. Try again.
  • And in some runs:
    Cluster <example-ecs-cluster> is processing a previous update. Wait for the cluster attachments to be in UPDATE_COMPLETE or UPDATE_FAILED state and try again.

Hypothesized behavior:

• There appears to be a hidden ordering or dependency between:
  • ECS cluster capacity provider attachments (ClusterCapacityProviders), and
  • The Capacity Provider’s autoScalingGroupProvider.managedScaling settings.
• When Pulumi issues an in-place update to the Capacity Provider, ECS may treat the cluster as still processing previous capacity provider attachment updates, causing the update to fail and resulting in UPDATE_FAILED on the capacity provider.

Manual AWS Console workaround (high level):

• In testing, the configuration change has been made to succeed by using the AWS Console and performing a multi-step sequence that temporarily adjusts capacity provider settings and/or cluster attachments before re-applying the desired managed scaling configuration.

• This behavior suggests that a different sequence of ECS API operations (or additional waits around cluster capacity provider attachment state) might allow the Pulumi AWS provider to perform these updates reliably without manual intervention.

• The issue has been reproduced with @pulumi/aws v6.81.0 and v7.10.0.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[pgavlin]

Does this repro consistently--i.e. does every attempt to update the capacity provider fail? the internal error referenced in the output is coming from the AWS API.