eks.Cluster fails on coredns and does not add NodeGroup

[jesusch]

Hi,

thank you for your work on this!!
I am trying to bootstrap an EKS cluster with a system nodegroup

1.

It always fails on the first run while creating the coredns addon

  aws:eks:Addon (devsandbox-coredns):
    error:   sdk-v2/provider2.go:566: sdk.helper_schema: waiting for EKS Add-On (devsandbox:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s): provider=aws@6.79.1
    error: 1 error occurred:
        * creating urn:pulumi:devsandbox::k8s::eks:index:Cluster$aws:eks/addon:Addon::devsandbox-coredns: 1 error occurred:
        * waiting for EKS Add-On (devsandbox:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)

which seems sane, as at this point there are no worker-nodes where coredns could run?

on the second it goes on, while the coredns addon remains in "Degraded" status

2.

The stack has created an autoscaling group but it is not added to the cluster?

    cluster_name = "mycluster"
    subnet_ids = get_eks_subnets() # list of subnet id's


    # Create the EKS Cluster
    cluster = eks.Cluster(cluster_name,
        name=cluster_name,
        authentication_mode=eks.AuthenticationMode.API,
        access_entries=get_access_entries(),
        # default_addons_to_remove=["kube-proxy", "coredns", "vpc-cni"],
        endpoint_private_access=True,
        endpoint_public_access=False,
        create_oidc_provider=True,
        kubernetes_service_ip_address_range="172.20.0.0/16",
        private_subnet_ids=subnet_ids,
        vpc_id=vpc_id,
        instance_type="t3.medium",
        desired_capacity=2,
        min_size=1,
        max_size=6,
        # Do not give worker nodes a public IP address
        node_associate_public_ip_address=False,
    )

output:

Updating (devsandbox):
     Type                                   Name                                            Status                         
 +   pulumi:pulumi:Stack                    k8s-devsandbox                                  **creating failed (1787s)**    
 +   ├─ aws:iam:Role                        eks-nodegroup-role-devsandbox                   created (1s)                   
 +   ├─ aws:iam:RolePolicyAttachment        eks-cni-policy-attachment                       created (0.71s)                
 +   ├─ aws:iam:RolePolicyAttachment        eks-worker-node-policy-attachment               created (1s)                   
 +   ├─ aws:iam:RolePolicyAttachment        ec2-container-registry-read-only-attachment     created (1s)                   
 +   ├─ aws:iam:RolePolicyAttachment        eks-nodegroup-policy-ssm-managed-instance-core  created (1s)                   
 +   └─ eks:index:Cluster                   devsandbox                                      created (748s)                 
 +      ├─ eks:index:ServiceRole            devsandbox-instanceRole                         created (8s)                   
 +      │  ├─ aws:iam:Role                  devsandbox-instanceRole-role                    created (2s)                   
 +      │  ├─ aws:iam:RolePolicyAttachment  devsandbox-instanceRole-e1b295bd                created (1s)                   
 +      │  ├─ aws:iam:RolePolicyAttachment  devsandbox-instanceRole-3eb088f2                created (1s)                   
 +      │  └─ aws:iam:RolePolicyAttachment  devsandbox-instanceRole-03516f97                created (1s)                   
 +      ├─ eks:index:ServiceRole            devsandbox-eksRole                              created (9s)                   
 +      │  ├─ aws:iam:Role                  devsandbox-eksRole-role                         created (1s)                   
 +      │  └─ aws:iam:RolePolicyAttachment  devsandbox-eksRole-4b490823                     created (2s)                   
 +      ├─ aws:ec2:SecurityGroup            devsandbox-eksClusterSecurityGroup              created (3s)                   
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksClusterInternetEgressRule         created (0.86s)                
 +      ├─ aws:iam:InstanceProfile          devsandbox-instanceProfile                      created (7s)                   
 +      ├─ aws:eks:Cluster                  devsandbox-eksCluster                           created (557s)                 
 +      ├─ pulumi:providers:kubernetes      devsandbox-eks-k8s                              created (0.52s)                
 +      ├─ aws:ec2:SecurityGroup            devsandbox-nodeSecurityGroup                    created (3s)                   
 +      ├─ aws:eks:AccessEntry              devsandbox-teamcity                             created (1s)                   
 +      ├─ aws:eks:AccessEntry              devsandbox-defaultNodeGroupInstanceRole         created (2s)                   
 +      ├─ aws:eks:AccessEntry              devsandbox-SecurityAudit                        created (5s)                   
 +      ├─ aws:eks:AccessEntry              devsandbox-Admin-NonEU                          created (3s)                   
 +      ├─ aws:iam:OpenIdConnectProvider    devsandbox-oidcProvider                         created (3s)                   
 +      ├─ aws:eks:AccessEntry              devsandbox-gitlab                               created (4s)                   
 +      ├─ aws:eks:Addon                    devsandbox-kube-proxy                           created (142s)                 
 +      ├─ aws:eks:Addon                    devsandbox-coredns                              **creating failed**            
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksNodeInternetEgressRule            created (1s)                   
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksNodeClusterIngressRule            created (4s)                   
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksClusterIngressRule                created (2s)                   
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksNodeIngressRule                   created (3s)                   
 +      ├─ aws:ec2:SecurityGroupRule        devsandbox-eksExtApiServerClusterIngressRule    created (3s)                   
 +      ├─ eks:index:VpcCniAddon            devsandbox-vpc-cni                              created (147s)                 
 +      │  └─ aws:eks:Addon                 devsandbox-vpc-cni                              created (139s)                 
 +      ├─ aws:ec2:LaunchTemplate           devsandbox-launchTemplate                       created (9s)                   
 +      └─ aws:autoscaling:Group            devsandbox                                      created (18s)                  

Diagnostics:
  aws:eks:Addon (devsandbox-coredns):
    error:   sdk-v2/provider2.go:566: sdk.helper_schema: waiting for EKS Add-On (devsandbox:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s): provider=aws@6.79.1
    error: 1 error occurred:
        * creating urn:pulumi:devsandbox::k8s::eks:index:Cluster$aws:eks/addon:Addon::devsandbox-coredns: 1 error occurred:
        * waiting for EKS Add-On (devsandbox:coredns) create: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)

  pulumi:pulumi:Stack (k8s-devsandbox):
    error: update failed
    error: Resource monitor has terminated, shutting down

Resources:
    + 37 created

Duration: 29m52s

➜  k8s git:(TVS-12394-fix-prodeu-kops) ✗ pulumi up -fy                                                     <aws:devsandbox>
warning: No value for configuration key "rds:postgres_instances". This is currently treated as an empty string `""`, but will be treated as `null` in a future version of pulumi.
Set the value to `""` to avoid this warning.
Updating (devsandbox):
     Type                          Name                                  Status           
     pulumi:pulumi:Stack           k8s-devsandbox                                         
     ├─ eks:index:Cluster          devsandbox                                             
 ~   │  └─ aws:eks:Addon           devsandbox-coredns                    updated (1s)     
 +   └─ aws:ec2:SecurityGroupRule  eks-control-plane-ingress-devsandbox  created (2s)     

Resources:
    + 1 created
    ~ 1 updated
    2 changes. 37 unchanged

Duration: 27s

[jesusch]

when I add coredns_addon_options=eks.CoreDnsAddonOptionsArgs(enabled=False) it runs until the end
But it still does not create a NodeGroup?

[jesusch]

pulumi.export("nodegroup", cluster.default_node_group)

Outputs:
  + nodegroup: {
      + auto_scaling_group        : {
          + arn                             : "arn:aws:autoscaling:eu-west-1:1234:autoScalingGroup:1e5f0a7c-533e-435e-9a26-15ea3ec23b28:autoScalingGroupName/devsandbox-c389f60"
          + availability_zone_distribution  : {
              + capacity_distribution_strategy: "balanced-best-effort"
            }
          + availability_zones              : [
          +     [0]: "eu-west-1a"
          +     [1]: "eu-west-1b"
          +     [2]: "eu-west-1c"
            ]
          + capacity_rebalance              : false
          + context                         : ""
          + default_cooldown                : 300
          + default_instance_warmup         : 0
          + desired_capacity                : 2
          + desired_capacity_type           : ""
          + enabled_metrics                 : <null>
          + force_delete                    : false
          + force_delete_warm_pool          : false
          + health_check_grace_period       : 300
          + health_check_type               : "EC2"
          + id                              : "devsandbox-c389f60"
          + ignore_failed_scaling_activities: false
          + initial_lifecycle_hooks         : []
          + instance_maintenance_policy     : <null>
          + instance_refresh                : {
              + preferences: {
                  + auto_rollback               : false
                  + checkpoint_delay            : ""
                  + instance_warmup             : ""
                  + max_healthy_percentage      : 100
                  + min_healthy_percentage      : 50
                  + scale_in_protected_instances: "Ignore"
                  + skip_matching               : false
                  + standby_instances           : "Ignore"
                }
              + strategy   : "Rolling"
            }
          + launch_configuration            : ""
          + launch_template                 : {
              + id     : "lt-0b9d5fbef4d5bb734"
              + name   : "devsandbox-launchTemplate-ad0b162"
              + version: "1"
            }
          + load_balancers                  : []
          + max_instance_lifetime           : 0
          + max_size                        : 6
          + metrics_granularity             : "1Minute"
          + min_elb_capacity                : <null>
          + min_size                        : 1
          + mixed_instances_policy          : <null>
          + name                            : "devsandbox-c389f60"
          + name_prefix                     : ""
          + placement_group                 : ""
          + predicted_capacity              : 0
          + protect_from_scale_in           : false
          + service_linked_role_arn         : "arn:aws:iam::1234:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
          + suspended_processes             : <null>
          + tags                            : [
          +     [0]: {
                  + key                : "Name"
                  + propagate_at_launch: true
                  + value              : "devsandbox-worker"
                }
          +     [1]: {
                  + key                : "kubernetes.io/cluster/devsandbox"
                  + propagate_at_launch: true
                  + value              : "owned"
                }
            ]
          + target_group_arns               : []
          + termination_policies            : <null>
          + traffic_sources                 : []
          + urn                             : "urn:pulumi:devsandbox::k8s::eks:index:Cluster$aws:autoscaling/group:Group::devsandbox"
          + vpc_zone_identifiers            : [
          +     [0]: "subnet-1234"
          +     [1]: "subnet-1234"
          +     [2]: "subnet-1234"
            ]
          + wait_for_capacity_timeout       : "10m"
          + wait_for_elb_capacity           : <null>
          + warm_pool                       : <null>
          + warm_pool_size                  : 0
        }
      + extra_node_security_groups: <null>
      + node_security_group       : {
          + arn                   : "arn:aws:ec2:eu-west-1:1234:security-group/sg-02654c2feae16b885"
          + description           : "Managed by Pulumi"
          + egress                : []
          + id                    : "sg-1234"
          + ingress               : []
          + name                  : "devsandbox-nodeSecurityGroup-37b484a"
          + name_prefix           : ""
          + owner_id              : "1234"
          + revoke_rules_on_delete: true
          + tags                  : {
              + Name                            : "devsandbox-nodeSecurityGroup"
              + kubernetes.io/cluster/devsandbox: "owned"
            }
          + tags_all              : {
              + Name                            : "devsandbox-nodeSecurityGroup"
              + kubernetes.io/cluster/devsandbox: "owned"
            }
          + urn                   : "urn:pulumi:devsandbox::k8s::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::devsandbox-nodeSecurityGroup"
          + vpc_id                : "vpc-asdf"
        }
    }

[blampe]

Hi @jesusch. Sorry you're running into this. It's hard to diagnose the problem without knowing more about how your VPC is configured. Make sure it's setup for the private cluster endpoint, and if you'd like more help please include all of the relevant code you're using to provision the cluster.