Validation for VPC Service Controls Policies during Terraform Plan Operation

[sannorw]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

OpenTofu 1.9.0
on amd64/linux

• provider registry.terraform.io/hashicorp/google v6.27.0

Affected Resource(s)

google_access_context_manager_service_perimeter_ingress_policy
google_access_context_manager_service_perimeter_egress_policy
google_access_context_manager_service_perimeter_dry_run_ingress_policy
google_access_context_manager_service_perimeter_dry_run_egress_policy

Terraform Configuration

There is no extensive validation of the policy content during a plan operation. Given the serious issues a misconfigured VPC Service Control Perimeter could produce, this is not appropriate.

In fact, we were having a major production issue, because a misconfigured ingress rule for the Cloud KMS service was blocking the access for many services to their encryption keys. While executing the plan operation, all was successfully validated and we were informed that the ingress policy resources will be replaced. On the apply operation, the old configuration was removed but the creation of the new one failed. This teared many of our productive services down and would have been avoided, if the validation of the resource during the plan operation was done more extensively.

Configuration snippet that was causing the issue ("serviceAccount:" prefix was missing):

    identities = [
      "serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com",
      "service-345678901234@gcp-sa-logging.iam.gserviceaccount.com",
    ]

The correct configuration:

    identities = [
      "serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com",
      "serviceAccount:service-345678901234@gcp-sa-logging.iam.gserviceaccount.com",
    ]

Full resource block of the non-working configuration:

resource "google_access_context_manager_service_perimete_ingress_policy" "kms" {
  title = "kms"
  # description
  perimeter = google_access_context_manager_service_perimeter.my_org.name
  ingress_from {
    identities = [
      "serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com",
      "service-345678901234@gcp-sa-logging.iam.gserviceaccount.com",
    ]
    sources {
      access_level = "*"
    }
  }
  ingress_to {
    resources = [local.project_kms]
    operations {
      service_name = "cloudkms.googleapis.com"
      method_selectors {
        method = "*"
      }
    }
  }
}

Debug Output

tofu plan output:

OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
OpenTofu will perform the following actions:
  # google_access_context_manager_service_perimeter_ingress_policy.kms must be replaced
-/+ resource "google_access_context_manager_service_perimeter_ingress_policy" "kms" {
      ~ access_policy_id = "accessPolicies/9876543210987" -> (known after apply)
      + etag             = (known after apply)
      ~ id               = "accessPolicies/9876543210987/servicePerimeters/my_org" -> (known after apply)
        # (2 unchanged attributes hidden)
      ~ ingress_from {
          ~ identities = [ # forces replacement
                "serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com",
              + "service-345678901234@gcp-sa-logging.iam.gserviceaccount.com",
            ]
            # (1 unchanged block hidden)
        }
        # (1 unchanged block hidden)
    }
Plan: 1 to add, 0 to change, 1 to destroy.

tofu apply output:

google_access_context_manager_service_perimeter_ingress_policy.kms: Destroying... [id=accessPolicies/9876543210987/servicePerimeters/my_org]
google_access_context_manager_service_perimeter_ingress_policy.kms: Destruction complete after 2s
google_access_context_manager_service_perimeter_ingress_policy.kms: Creating...
╷
│ Error: Error creating ServicePerimeterIngressPolicy: googleapi: Error 400: 'service-345678901234@gcp-sa-logging.iam.gserviceaccount.com' doesn't match expected format.
│ 
│   with google_access_context_manager_service_perimeter_ingress_policy.kms,
│   on ingress-policies.tf line 112, in resource "google_access_context_manager_service_perimeter_ingress_policy" "kms":
│  112: resource "google_access_context_manager_service_perimeter_ingress_policy" "kms" {
│ 
╵

Expected Behavior

A failed plan operation due to a failed validation of the resource block values.

Actual Behavior

Successful plan operation, but failing apply operation afterwards

Steps to reproduce

1. Create a resource "google_access_context_manager_service_perimete_ingress_policy" with correctly specified service accounts
2. Add a service account, without ServiceAccount: prefix
3. Run a plan operation -> successful
4. Run an apply operation -> fails

Important Factoids

No response

References

No response

[Charlesleonius]

We will look into adding some common sense validations for our fields. For resources such as google_access_context_manager_service_perimeter_ingress_policy, you can use https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#create_before_destroy to ensure that new / modified ingress or egress rules are added before the old ones are destroyed which may help avoid such instances.