State backend replication fails: replication role denied kms:Decrypt since 1.20.1

[yousefdebaz-fivexlio]

Terraform Version & Prov: See below.

AFT Version:
1.21.1

Introduced in 1.20.1. 1.20.0 and earlier are not affected.

Terraform Version & Provider Versions

terraform version

Terraform v1.14.8
on darwin_arm64

terraform providers

provider[registry.terraform.io/hashicorp/aws] 6.56.0
provider[registry.terraform.io/hashicorp/archive] 2.8.0
provider[registry.terraform.io/hashicorp/external] 2.4.0
provider[registry.terraform.io/hashicorp/local] 2.9.0
provider[registry.terraform.io/hashicorp/random] 3.9.0
provider[registry.terraform.io/hashicorp/time] 0.14.0
provider[registry.terraform.io/hashicorp/tls] 4.3.0

terraform_version for the AFT pipelines is set to 1.7.5.

Bug Description

Cross-region replication of the Terraform state backend bucket never succeeds. The aft-s3-terraform-backend-replication role is denied kms:Decrypt on the primary region backend key, so nothing is ever replicated into aft-backend-<account>-secondary-region.

The cause is a mismatch between two things in modules/aft-backend/main.tf:

1. Since 1.20.1 both backend buckets are created with an S3 Bucket Key, bucket_key_enabled = true at lines 101 and 136.
2. The replication role policy still conditions its KMS grants on the object ARN as the encryption context, lines 225-270:

"Condition": {
    "StringLike": {
        "kms:ViaService": "s3.${var.primary_region}.amazonaws.com",
        "kms:EncryptionContext:aws:s3:arn": [
            "${aws_s3_bucket.primary-backend-bucket.arn}/*"
        ]
    }
}

When an S3 Bucket Key is in use the KMS encryption context is the bucket ARN rather than the object ARN, and IAM policies have to be updated to match. See Configuring your bucket to use an S3 Bucket Key with SSE-KMS for new objects, which states that when a bucket key is enabled on the source or destination bucket the encryption context becomes the bucket ARN, and that IAM policies need to use the bucket ARN for the encryption context.

arn:aws:s3:::aft-backend-<account>-primary-region therefore never matches arn:aws:s3:::aft-backend-<account>-primary-region/*, the condition never evaluates true, and the grant is inert. All three KMS statements in aws_iam_policy.replication have this problem: kms:Decrypt and kms:Encrypt on the primary region key, and kms:Encrypt on the secondary region key.

This fails quietly. terraform apply succeeds, the replication configuration looks correct, and no AFT pipeline reports an error. The only symptoms are AccessDenied entries in CloudTrail and an empty secondary region bucket, so losing the DR copy of the state is easy to miss.

Bucket keys appear to have been enabled in response to #339.

To Reproduce

1. Deploy AFT 1.20.1 or later with tf_backend_secondary_region set.
2. Run any pipeline that writes Terraform state, so at least one object lands in aft-backend-<account>-primary-region.
3. List aft-backend-<account>-secondary-region. It is empty.
4. Check the replication status of a source object:
   aws s3api head-object --bucket aft-backend-<account>-primary-region --key <key> --query ReplicationStatus
   returns FAILED.
5. CloudTrail in the AFT management account shows kms:Decrypt AccessDenied events for aft-s3-terraform-backend-replication.

Expected behavior

Objects written to aft-backend-<account>-primary-region replicate to aft-backend-<account>-secondary-region.

Related Logs

CloudTrail event from the AFT management account, account and key IDs redacted:

arn:aws:sts::<AFT-ACCOUNT-ID>:assumed-role/aft-s3-terraform-backend-replication/s3-replication
called Decrypt but failed due to AccessDenied

Error message:
User: arn:aws:sts::<AFT-ACCOUNT-ID>:assumed-role/aft-s3-terraform-backend-replication/s3-replication
is not authorized to perform: kms:Decrypt on resource:
arn:aws:kms:us-east-1:<AFT-ACCOUNT-ID>:key/<BACKEND-KMS-KEY-ID>
because no identity-based policy allows the kms:Decrypt action

The key in that message is the one behind alias/aft-backend-<account>-kms-key in the primary region, that is aws_kms_key.encrypt-primary-region.

Additional context

Suggested fix: use the bucket ARN for the encryption context in all three KMS statements of aws_iam_policy.replication. Accepting both forms keeps the policy working if bucket keys are ever disabled again:

"kms:EncryptionContext:aws:s3:arn": [
    "${aws_s3_bucket.primary-backend-bucket.arn}",
    "${aws_s3_bucket.primary-backend-bucket.arn}/*"
]

kms:EncryptionContext:aws:s3:arn is a single valued condition key, so the list is evaluated as an OR.

Two related points:

• The secondary region statement may also need kms:GenerateDataKey alongside kms:Encrypt, since the destination bucket has a bucket key enabled as well.
• Fixing the policy does not backfill. Only new and re-uploaded objects replicate, so operators who have been running 1.20.1 or later will need S3 Batch Replication to copy the existing state objects across.

For anyone hitting this before a release, the workaround is a second inline policy on the role, which sits alongside the AFT managed one rather than replacing it:

resource "aws_iam_role_policy" "replication_bucket_key_encryption_context" {
  name = "aft-s3-terraform-backend-replication-bucket-key-encryption-context"
  role = "aft-s3-terraform-backend-replication"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["kms:Decrypt", "kms:Encrypt"]
        Resource = [local.primary_backend_key_arn]
        Condition = {
          StringEquals = { "kms:ViaService" = "s3.${local.primary_region}.amazonaws.com" }
          StringLike = {
            "kms:EncryptionContext:aws:s3:arn" = [
              local.primary_backend_bucket_arn,
              "${local.primary_backend_bucket_arn}/*",
            ]
          }
        }
      },
      {
        Effect   = "Allow"
        Action   = ["kms:Encrypt", "kms:GenerateDataKey"]
        Resource = [local.secondary_backend_key_arn]
        Condition = {
          StringEquals = { "kms:ViaService" = "s3.${local.secondary_region}.amazonaws.com" }
          StringLike = {
            "kms:EncryptionContext:aws:s3:arn" = [
              local.secondary_backend_bucket_arn,
              "${local.secondary_backend_bucket_arn}/*",
            ]
          }
        }
      },
    ]
  })
}

[yousefdebaz-fivexlio]

Related to #339, which was reopened on the grounds that the S3 Bucket Key work was incomplete. This looks like the remaining piece of it.

One correction for the record: bucket_key_enabled is absent from every .tf file at the 1.20.0 tag and appears six times at 1.20.1, so the bucket keys shipped in 1.20.1 rather than 1.20.0. 1.20.0 and earlier are unaffected by this bug.

Scope of what the bucket key change touched, as of 1.21.1:

• The bucket side is complete. All six aws_s3_bucket_server_side_encryption_configuration resources in the module set bucket_key_enabled = true: both backend buckets and the backend access logs bucket in modules/aft-backend/main.tf, the customizations pipeline bucket in modules/aft-customizations/s3.tf, and the two logging buckets in modules/aft-feature-options/s3.tf.
• kms:EncryptionContext:aws:s3:arn occurs in only three places in the whole tree, and all three are the KMS statements in aws_iam_policy.replication in modules/aft-backend/main.tf (lines 231, 248, 265). Every other consumer of these keys either has no encryption context condition or relies on the default key policy.

So this policy appears to be the only one the bucket key change invalidated, which should keep the fix bounded to those three statements.