Description
When re-registering an OU with a large number of accounts (100+), AFT's aft-account-provisioning-framework Step Function triggers aft-create-pipeline CodeBuild builds for all accounts simultaneously. Because the CodeBuild project runs in a VPC, each build requires ENI creation via the EC2 API. This causes RequestLimitExceeded errors on CreateNetworkInterface, as the EC2 API rate limit (bucket max: 100, refill: 5/sec) is not adjustable.
All builds fail during the QUEUED phase with:
{
"Contexts": [{"Message": "Unexpected EC2 error: RequestLimitExceeded", "StatusCode": "VPC_CLIENT_ERROR"}],
"PhaseStatus": "CLIENT_ERROR",
"PhaseType": "QUEUED"
}
The affected accounts then show "internal failure" in Control Tower and require manual re-triggering.
Proposed Solution
Add a configurable concurrent_build_limit attribute to the aws_codebuild_project.aft_create_pipeline resource, exposed as an input variable (e.g., aft_create_pipeline_concurrent_build_limit) with a sensible default (e.g., 20–30).
The Terraform resource already supports this:
resource "aws_codebuild_project" "aft_create_pipeline" {
# ...existing configuration...
concurrent_build_limit = var.aft_create_pipeline_concurrent_build_limit
}Reference: Terraform aws_codebuild_project docs — concurrent_build_limit
(Optional) Specify a maximum number of concurrent builds for the project. The value specified must be greater than 0 and less than the account concurrent running builds limit.
This would allow builds to queue in CodeBuild rather than all attempting to acquire VPC ENIs simultaneously, staying within the non-adjustable EC2 API rate limits.
Impact Without This Change
- Any OU re-registration or bulk account operation with >100 accounts will reliably fail
- The EC2 CreateNetworkInterface rate limits (100 bucket max, 5/sec refill) are not adjustable per AWS Service Quotas
- There is no workaround available to AFT consumers short of forking the module
- Failed accounts require manual re-triggering, which is operationally burdensome at scale
Environment
- AFT deployed in VPC mode (CodeBuild uses VPC configuration with subnets/security groups)
- 134 accounts in the re-registered OU
- All builds submitted within the same second, exhausting the EC2 API token bucket
Suggested Default
A default of 20 or 30 would prevent EC2 API throttling while still allowing reasonable parallelism. This could also be applied to aft_account_provisioning_customizations_pipeline and other VPC-mode CodeBuild projects for consistency.
Description
When re-registering an OU with a large number of accounts (100+), AFT's aft-account-provisioning-framework Step Function triggers aft-create-pipeline CodeBuild builds for all accounts simultaneously. Because the CodeBuild project runs in a VPC, each build requires ENI creation via the EC2 API. This causes RequestLimitExceeded errors on CreateNetworkInterface, as the EC2 API rate limit (bucket max: 100, refill: 5/sec) is not adjustable.
All builds fail during the QUEUED phase with:
The affected accounts then show "internal failure" in Control Tower and require manual re-triggering.
Proposed Solution
Add a configurable concurrent_build_limit attribute to the aws_codebuild_project.aft_create_pipeline resource, exposed as an input variable (e.g., aft_create_pipeline_concurrent_build_limit) with a sensible default (e.g., 20–30).
The Terraform resource already supports this:
Reference: Terraform aws_codebuild_project docs — concurrent_build_limit
(Optional) Specify a maximum number of concurrent builds for the project. The value specified must be greater than 0 and less than the account concurrent running builds limit.
This would allow builds to queue in CodeBuild rather than all attempting to acquire VPC ENIs simultaneously, staying within the non-adjustable EC2 API rate limits.
Impact Without This Change
Environment
Suggested Default
A default of 20 or 30 would prevent EC2 API throttling while still allowing reasonable parallelism. This could also be applied to aft_account_provisioning_customizations_pipeline and other VPC-mode CodeBuild projects for consistency.