Skip to content

Bug: GitLab webhooks not created for CodePipeline due to 'pending' CodeConnection status on initial apply #589

@devopstracr

Description

@devopstracr

Terraform Version & Prov:

AFT Version:
1.15.1 (From module path: github.com/aws-ia/terraform-aws-control_tower_account_factory version 1.15.1)

Terraform Version & Provider Versions
Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version

Terraform v1.3.5
on darwin_arm64

terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] >= 5.89.0
├── provider[registry.terraform.io/hashicorp/local]
├── module.aft_ssm_parameters
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   └── provider[registry.terraform.io/hashicorp/random]
├── module.packaging
│   └── provider[registry.terraform.io/hashicorp/archive]
├── module.aft_lambda_layer
│   ├── provider[registry.terraform.io/hashicorp/local]
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   └── provider[registry.terraform.io/hashicorp/random]
├── module.aft_feature_options
│   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
├── module.aft_iam_roles
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
│   ├── module.aft_exec_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.aft_service_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.audit_exec_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.audit_service_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.ct_management_exec_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.ct_management_service_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── module.log_archive_exec_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   └── module.log_archive_service_role
│       └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
├── module.aft_backend
│   └── provider[registry.terraform.io/hashicorp/aws] >= 4.27.0
├── module.aft_customizations
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
│   ├── provider[registry.terraform.io/hashicorp/time]
│   └── provider[registry.terraform.io/hashicorp/local]
├── module.aft_account_provisioning_framework
│   └── provider[registry.terraform.io/hashicorp/aws] >= 5.11.0, < 6.0.0
├── module.aft_account_request_framework
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.9.0
│   └── provider[registry.terraform.io/hashicorp/time]
└── module.aft_code_repositories
    ├── provider[registry.terraform.io/hashicorp/aws] >= 5.84.0, < 6.0.0
    ├── provider[registry.terraform.io/hashicorp/time]
    └── provider[registry.terraform.io/hashicorp/local]

Providers required by state:

    provider[registry.terraform.io/hashicorp/archive]

    provider[registry.terraform.io/hashicorp/local]

    provider[registry.terraform.io/hashicorp/time]

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/hashicorp/random]

Bug Description
When using the AFT module to create new accounts, the corresponding CodePipelines for GitLab repositories are not setting up their webhooks correctly on the first terraform apply.
This appears to be a timing or dependency issue. The module creates (or is used to create) both the aws_codeconnections_connection and the aws_codepipeline in the same apply. However, the connection resource enters a Pending state, requiring manual intervention in the GitLab UI to move it to an Available state.
Because the connection is not Available when the aws_codepipeline resource is created, the pipeline resource fails (silently) to create the required webhook in GitLab.

To Reproduce
Steps to reproduce the behavior:

  1. Configure the AFT module to vend an account that creates a new CodePipeline sourced from a GitLab repository (which requires a new aws_codeconnections_connection).
  2. Run terraform apply for the first time.
  3. Observe that both the aws_codeconnections_connection and aws_codepipeline resources are created in AWS. The connection is in a Pending state.
  4. Go to the AWS Console (Developer Tools > CodeConnections) and complete the manual handshake to move the connection from Pending to Available.
  5. Go to the target GitLab repository's "Webhooks" section.
    See that no webhook has been created by AWS CodePipeline.

Expected behavior
The module should handle this "pending" state, or the documentation should clearly state the required manual workaround. Ideally, the aws_codepipeline resource creation would wait until the connection's status is Available, not just until the resource exists.

Related Logs
There are no explicit error messages from the terraform apply. The failure is silent. The only symptom is the lack of a webhook in the GitLab repository, which prevents the pipeline from triggering automatically.

Additional context
This is a "Day 1" provisioning problem. The aws_codepipeline resource doesn't seem to retry webhook creation or react to the aws_codeconnections_connection status changing from Pending to Available after the fact. It only attempts to create the webhook when the pipeline resource itself is first created.

Manual Workaround: The following manual process successfully creates the webhooks:

  1. Run the initial terraform apply.
  2. Go to the GitLab UI and manually approve/configure the CodeConnection until it reports as Available in the AWS Console.
  3. Manually delete the newly created CodePipeline(s) via the AWS Console.
  4. Run terraform apply a second time.
  5. Terraform detects the pipelines are missing and recreates them.
  6. Since the CodeConnection is now Available, the pipeline creation succeeds, and the webhooks are correctly created in GitLab.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpending investigationIssue needs further investigation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions