`google_container_cluster` unable to create default node pool when `workload_identity_config` is set

[phardy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.11.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v6.27.0

Affected Resource(s)

google_container_cluster

Terraform Configuration

This is based on the example usage in the resource documentation, with some necessary changes for my test environment. It uses a small shared subnet, and node_locations and default_max_pods_per_node are tuned down to match.

data "google_compute_subnetwork" "cna_subnet" {
  name    = "gke-cluster-confluence-us-east4-05"
  project = data.google_compute_network.cna_network.project
}

data "google_project" "project" {}

resource "google_service_account" "workload_identity_test" {
  account_id   = "workload-identity-test"
  display_name = "Workload Identity Test Service Account"
}

resource "google_container_cluster" "workload_identity_test" {
  name     = "workload-identity-test"
  location = "us-east4"
  node_locations = [
    "us-east4-b",
    "us-east4-c",
  ]

  initial_node_count = 1

  node_config {
    service_account = google_service_account.workload_identity_test.email
    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]
  }

  default_max_pods_per_node = 56

  network    = data.google_compute_network.cna_network.id
  subnetwork = data.google_compute_subnetwork.cna_subnet.id

  ip_allocation_policy {
    cluster_secondary_range_name = data.google_compute_subnetwork.cna_subnet.secondary_ip_range[0].range_name
  }

  enterprise_config {
    desired_tier = "ENTERPRISE"
  }

  workload_identity_config {
    workload_pool = "${data.google_project.project.project_id}.svc.id.goog"
  }

  timeouts {
    create = "120m"
    update = "120m"
    read   = "120m"
  }
}

Debug Output

https://gist.github.com/phardy/d1ee5b1cdf4ae5d8ec17f2c3b5aa147b

Expected Behavior

I expect a new cluster to be created, with the default node pool still attached. I expect this to take some time - experience testing similar configurations shows at least 30 minutes, and more to add a workload pool afterwards.

Actual Behavior

The terraform apply fails after ~40 minutes, despite the 120m timeouts specified in the resource, with this error:

Error: Error waiting for creating GKE cluster: All cluster resources were brought up, but: 2 nodes out of 2 are unhealthy.

Inspecting the cluster in the GCS console shows the same error reported by the console. Inspecting the default node pool shows it reporting all nodes OK. Running a terraform plan at this point indicates the existing cluster is tainted, and proposes destroying it and creating a new cluster.

Steps to reproduce

1. terraform apply

Important Factoids

As mentioned with the terraform configuration, I'm using a small shared subnet. The primary IPv4 range for this subnet is a /28, and it has a single secondary IPv4 /24 range.

I've tested an identical configuration without the workload_identity_config block, which created successfully. And then added the workload_identity_config block, making the final config identical to what I've pasted here. Terraform successfully modifies the cluster enabling workload, although this takes approximately 30 minutes to apply.

My initial attempts at creating this were using a separately managed node pool. However this also fails, my understanding reading the documentation and experimenting is that the provider creates the cluster with a default pool and then deletes it. And this initial node pool creation is unsuccessful.

I've also attempted creating this with the default service account (omitting service_account and oauth_scopes from the node_config block). This makes no change to the Actual Behaviour.

References

No response

b/409662502

[ggtisc]

Hi @phardy

Could you share an example of the values you are using, because after trying to replicate this issue with the below code everything works as expected without errors. You can just specify which values need to be changed (and their values)

data "google_project" "project" {}

resource "google_compute_network" "vpc_22168" {
  name                    = "vpc-22168"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "subnet_22168" {
  name          = "subnet-22168"
  ip_cidr_range = "10.2.0.0/16"
  network       = google_compute_network.vpc_22168.id
  region        = "us-central1"

  secondary_ip_range {
    range_name    = "second-ip-range-22168"
    ip_cidr_range = "192.168.10.0/24"
  }
}

resource "google_service_account" "service_account_22168" {
  account_id   = "service-account-22168"
  display_name = "service_account_22168"
}

resource "google_container_cluster" "container_cluster_22168" {
  name                      = "container-cluster-22168"
  location                  = "us-central1"
  initial_node_count        = 1
  default_max_pods_per_node = 56
  network                   = google_compute_network.vpc_22168.id
  subnetwork                = google_compute_subnetwork.subnet_22168.id

  node_locations = [
    "us-central1-b",
    "us-central1-c"
  ]

  node_config {
    service_account = google_service_account.service_account_22168.email

    oauth_scopes = [
      "https://www.googleapis.com/auth/cloud-platform"
    ]
  }

  ip_allocation_policy {
    cluster_secondary_range_name = google_compute_subnetwork.subnet_22168.secondary_ip_range[0].range_name
  }

  enterprise_config {
    desired_tier = "ENTERPRISE"
  }

  workload_identity_config {
    workload_pool = "${data.google_project.project.project_id}.svc.id.goog"
  }

  timeouts {
    create = "120m"
    update = "120m"
    read   = "120m"
  }
}

[phardy]

Thanks for looking in to this @ggtisc , and apologies for the long delay getting back to you.

There's a couple of differences, firstly I've only been given a /28 subnet, with a /24 secondary range. I'm using a data resource to read this in, but putting the CIDRs I'm working with in to your google_compute_subnetwork resource looks like this.

resource "google_compute_subnetwork" "subnet_22168" {
  name          = "subnet-22168"
  ip_cidr_range = "10.186.96.0/28"
  network       = google_compute_network.vpc_22168.id
  region        = "us-central1"

  secondary_ip_range {
    range_name    = "second-ip-range-22168"
    ip_cidr_range = "100.64.48.0/24"
  }
}

The other key difference is that the google_compute_network and google_compute_subnetwork I'm working with are provided via shared VPC, and exist in a different project. I'm not managing them in my code, they were created on my behalf and shared to the project that I'm working in.

After reading your reply, though, I realised there was one important piece of testing that I hadn't yet done. So I've been attempting to replicate this problem by creating the cluster manually in the Google Cloud console. And I can confirm that I'm seeing the same behaviour there. I'm not able to manually create a cluster with Workload Identity enabled. But I can create the cluster with it disabled, and then turn it on.

As such, I'm becoming convinced this isn't a problem with the provider but something else, possibly unique to my environment. I'm happy to close this off for now, and thanks again for your time looking in to it.

[ggtisc]

Thanks for clarifying. Are you using Google beta provider, or just normal?

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
      source  = "hashicorp/google-beta" # Are you using this?
      version = "6.29.0"
    }
  }
}

[phardy]

Definitely using the normal provider, my last tests were with 6.27 but I'll confirm again with 6.29.

[ggtisc]

After many tries I can't get any errors or bugs. i'm forwarding this issue for a deep review