Merge pull request #109 from rancherfederal/new-updates

Provider and policy updates

Author: adamacosta

README.md

@@ -153,15 +153,15 @@ Optional policies have the option of being created by default, but are specified
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3 |
-| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.6, <= 5.22 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.32 |
 | <a name="requirement_cloudinit"></a> [cloudinit](#requirement\_cloudinit) | >= 2 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 3 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
-| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.6, <= 5.22 |
+| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 5.32 |
 | <a name="provider_cloudinit"></a> [cloudinit](#provider\_cloudinit) | >= 2 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 3 |
 
@@ -205,11 +205,11 @@ Optional policies have the option of being created by default, but are specified
 | <a name="input_ami"></a> [ami](#input\_ami) | Server pool ami | `string` | n/a | yes |
 | <a name="input_associate_public_ip_address"></a> [associate\_public\_ip\_address](#input\_associate\_public\_ip\_address) | n/a | `bool` | `null` | no |
 | <a name="input_awscli_url"></a> [awscli\_url](#input\_awscli\_url) | URL for awscli zip file | `string` | `"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"` | no |
-| <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | Server pool block device mapping configuration | `map(string)` | <pre>{<br>  "encrypted": false,<br>  "size": 30<br>}</pre> | no |
+| <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | Server pool block device mapping configuration | `map(string)` | <pre>{<br/>  "size": "30"<br/>}</pre> | no |
 | <a name="input_ccm_external"></a> [ccm\_external](#input\_ccm\_external) | Set kubelet arg 'cloud-provider-name' value to 'external'.  Requires manual install of CCM. | `bool` | `false` | no |
-| <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Name of the rkegov cluster to create | `string` | n/a | yes |
+| <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Name of the rke2 cluster to create | `string` | n/a | yes |
 | <a name="input_controlplane_access_logs_bucket"></a> [controlplane\_access\_logs\_bucket](#input\_controlplane\_access\_logs\_bucket) | Bucket name for logging requests to control plane load balancer | `string` | `"disabled"` | no |
-| <a name="input_controlplane_allowed_cidrs"></a> [controlplane\_allowed\_cidrs](#input\_controlplane\_allowed\_cidrs) | Server pool security group allowed cidr ranges | `list(string)` | <pre>[<br>  "0.0.0.0/0"<br>]</pre> | no |
+| <a name="input_controlplane_allowed_cidrs"></a> [controlplane\_allowed\_cidrs](#input\_controlplane\_allowed\_cidrs) | Server pool security group allowed cidr ranges | `list(string)` | <pre>[<br/>  "0.0.0.0/0"<br/>]</pre> | no |
 | <a name="input_controlplane_enable_cross_zone_load_balancing"></a> [controlplane\_enable\_cross\_zone\_load\_balancing](#input\_controlplane\_enable\_cross\_zone\_load\_balancing) | Toggle between controlplane cross zone load balancing | `bool` | `true` | no |
 | <a name="input_controlplane_internal"></a> [controlplane\_internal](#input\_controlplane\_internal) | Toggle between public or private control plane load balancer | `bool` | `true` | no |
 | <a name="input_create_acl"></a> [create\_acl](#input\_create\_acl) | Toggle creation of ACL for statestore bucket | `bool` | `true` | no |
@@ -223,7 +223,7 @@ Optional policies have the option of being created by default, but are specified
 | <a name="input_iam_permissions_boundary"></a> [iam\_permissions\_boundary](#input\_iam\_permissions\_boundary) | If provided, the IAM role created for the servers will be created with this permissions boundary attached. | `string` | `null` | no |
 | <a name="input_instance_type"></a> [instance\_type](#input\_instance\_type) | Server pool instance type | `string` | `"t3a.medium"` | no |
 | <a name="input_lb_subnets"></a> [lb\_subnets](#input\_lb\_subnets) | List of subnet IDs to create load balancer in | `list(string)` | `null` | no |
-| <a name="input_metadata_options"></a> [metadata\_options](#input\_metadata\_options) | Instance Metadata Options | `map(any)` | <pre>{<br>  "http_endpoint": "enabled",<br>  "http_put_response_hop_limit": 2,<br>  "http_tokens": "required",<br>  "instance_metadata_tags": "disabled"<br>}</pre> | no |
+| <a name="input_metadata_options"></a> [metadata\_options](#input\_metadata\_options) | Instance Metadata Options | `map(any)` | <pre>{<br/>  "http_endpoint": "enabled",<br/>  "http_put_response_hop_limit": 3,<br/>  "http_tokens": "required",<br/>  "instance_metadata_tags": "disabled"<br/>}</pre> | no |
 | <a name="input_post_userdata"></a> [post\_userdata](#input\_post\_userdata) | Custom userdata to run immediately after rke2 node attempts to join cluster | `string` | `""` | no |
 | <a name="input_pre_userdata"></a> [pre\_userdata](#input\_pre\_userdata) | Custom userdata to run immediately before rke2 node attempts to join cluster, after required rke2, dependencies are installed | `string` | `""` | no |
 | <a name="input_rke2_channel"></a> [rke2\_channel](#input\_rke2\_channel) | Channel to use for RKE2 server nodepool | `string` | `null` | no |
@@ -238,7 +238,7 @@ Optional policies have the option of being created by default, but are specified
 | <a name="input_subnets"></a> [subnets](#input\_subnets) | List of subnet IDs to create nodes in | `list(string)` | n/a | yes |
 | <a name="input_suspended_processes"></a> [suspended\_processes](#input\_suspended\_processes) | List of processes to suspend in the autoscaling service | `list(string)` | `[]` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | Map of tags to add to all resources created | `map(string)` | `{}` | no |
-| <a name="input_termination_policies"></a> [termination\_policies](#input\_termination\_policies) | List of policies to decide how the instances in the Auto Scaling Group should be terminated | `list(string)` | <pre>[<br>  "Default"<br>]</pre> | no |
+| <a name="input_termination_policies"></a> [termination\_policies](#input\_termination\_policies) | List of policies to decide how the instances in the Auto Scaling Group should be terminated | `list(string)` | <pre>[<br/>  "Default"<br/>]</pre> | no |
 | <a name="input_unique_suffix"></a> [unique\_suffix](#input\_unique\_suffix) | Enables/disables generation of a unique suffix to cluster name | `bool` | `true` | no |
 | <a name="input_unzip_rpm_url"></a> [unzip\_rpm\_url](#input\_unzip\_rpm\_url) | URL path to unzip rpm | `string` | `""` | no |
 | <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | VPC ID to create resources in | `string` | n/a | yes |

data.tf

@@ -80,7 +80,7 @@ data "aws_iam_policy_document" "aws_required" {
   }
 }
 
-# Required IAM Policy for AWS CCM
+# Required IAM policy for AWS CCM (https://cloud-provider-aws.sigs.k8s.io/prerequisites/#iam-policies)
 data "aws_iam_policy_document" "aws_ccm" {
   count = var.iam_instance_profile == "" && var.enable_ccm ? 1 : 0
 
@@ -91,13 +91,13 @@ data "aws_iam_policy_document" "aws_ccm" {
       "autoscaling:DescribeAutoScalingGroups",
       "autoscaling:DescribeLaunchConfigurations",
       "autoscaling:DescribeTags",
-      "autoscaling:DescribeAutoScalingInstances",
       "ec2:DescribeInstances",
       "ec2:DescribeRegions",
       "ec2:DescribeRouteTables",
       "ec2:DescribeSecurityGroups",
       "ec2:DescribeSubnets",
       "ec2:DescribeVolumes",
+      "ec2:DescribeAvailabilityZones",
       "ec2:CreateSecurityGroup",
       "ec2:CreateTags",
       "ec2:CreateVolume",
@@ -112,6 +112,7 @@ data "aws_iam_policy_document" "aws_ccm" {
       "ec2:DetachVolume",
       "ec2:RevokeSecurityGroupIngress",
       "ec2:DescribeVpcs",
+      "ec2:DescribeInstanceTopology",
       "elasticloadbalancing:AddTags",
       "elasticloadbalancing:AttachLoadBalancerToSubnets",
       "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
@@ -148,6 +149,8 @@ data "aws_iam_policy_document" "aws_ccm" {
   }
 }
 
+# Required IAM policy for AWS Cluster Autoscaler
+# (https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#full-cluster-autoscaler-features-policy-recommended)
 data "aws_iam_policy_document" "aws_autoscaler" {
   count = var.enable_autoscaler ? 1 : 0
 
@@ -157,9 +160,11 @@ data "aws_iam_policy_document" "aws_autoscaler" {
       "autoscaling:DescribeAutoScalingInstances",
       "autoscaling:DescribeLaunchConfigurations",
       "autoscaling:DescribeScalingActivities",
-      "autoscaling:DescribeTags",
+      "ec2:DescribeImages",
       "ec2:DescribeInstanceTypes",
-      "ec2:DescribeLaunchTemplateVersions"
+      "ec2:DescribeLaunchTemplateVersions",
+      "ec2:GetInstanceTypesFromInstanceRequirements",
+      "eks:DescribeNodegroup"
     ]
     effect    = "Allow"
     resources = ["*"]
@@ -168,10 +173,7 @@ data "aws_iam_policy_document" "aws_autoscaler" {
   statement {
     actions = [
       "autoscaling:SetDesiredCapacity",
-      "autoscaling:TerminateInstanceInAutoScalingGroup",
-      "ec2:DescribeImages",
-      "ec2:GetInstanceTypesFromInstanceRequirements",
-      "eks:DescribeNodegroup"
+      "autoscaling:TerminateInstanceInAutoScalingGroup"
     ]
     effect    = "Allow"
     resources = ["*"]

examples/cloud-enabled/README.md

@@ -25,13 +25,14 @@ helm install aws-cloud-controller-manager aws-cloud-controller-manager/aws-cloud
 
 ## `cluster-autoscaler`
 
-Match region to your actual region, but it is `us-gov-west-1` in this example.
+Match region to your actual region, but it is `us-east-2` in this example. The cluster name also includes a UID suffix that will be created by Terraform at apply time, so change that to yours as well. The version here is coupled to the Kubernetes version. You can run `helm search repo -l autoscaler/cluster-autoscaler` to see all charts and the corresponding app version. Here, we grab the chart version that matches app version 1.33.0 because we are using `rke2` 1.33.
 
 ```sh
 helm repo add autoscaler https://kubernetes.github.io/autoscaler
 helm repo update
 helm install autoscaler autoscaler/cluster-autoscaler \
   --namespace kube-system \
-  --set autoDiscovery.clusterName=cloud-enabled-zjl \
-  --set awsRegion=us-gov-west-1
+  --version 9.51.0 \
+  --set autoDiscovery.clusterName=cloud-enabled-dyh \
+  --set awsRegion=us-east-2
 ```

examples/cloud-enabled/main.tf

@@ -6,11 +6,13 @@ provider "aws" {
 }
 
 locals {
+  ami_prefix   = "RHEL-9"
+  aws_region   = "us-east-2"
+  cidr         = "10.80.0.0/16"
   cluster_name = "cloud-enabled"
-  aws_region   = "us-gov-west-1"
-  cidr         = "10.88.0.0/16"
+
   ssh_allowed_cidrs = [
-    "0.0.0.0/0"
+    "76.185.97.220/32"
   ]
 
   tags = {
@@ -19,13 +21,13 @@ locals {
   }
 }
 
-data "aws_ami" "rhel8" {
+data "aws_ami" "server" {
   most_recent = true
-  owners      = ["219670896067"] # owner is specific to aws gov cloud
+  owners      = ["amazon"]
 
   filter {
     name   = "name"
-    values = ["RHEL-8*"]
+    values = ["${local.ami_prefix}*"]
   }
 
   filter {
@@ -56,8 +58,8 @@ module "vpc" {
   cidr = local.cidr
 
   azs             = ["${local.aws_region}a", "${local.aws_region}b", "${local.aws_region}c"]
-  public_subnets  = [cidrsubnet(local.cidr, 8, 1), cidrsubnet(local.cidr, 8, 2), cidrsubnet(local.cidr, 8, 3)]
-  private_subnets = [cidrsubnet(local.cidr, 8, 101), cidrsubnet(local.cidr, 8, 102), cidrsubnet(local.cidr, 8, 103)]
+  public_subnets  = [cidrsubnet(local.cidr, 8, 0), cidrsubnet(local.cidr, 8, 1), cidrsubnet(local.cidr, 8, 2)]
+  private_subnets = [cidrsubnet(local.cidr, 8, 10), cidrsubnet(local.cidr, 8, 11), cidrsubnet(local.cidr, 8, 12)]
 
   enable_nat_gateway   = true
   single_nat_gateway   = true
@@ -96,19 +98,19 @@ module "rke2" {
   vpc_id       = module.vpc.vpc_id
   subnets      = module.vpc.public_subnets # Note: Public subnets used for demo purposes, this is not recommended in production
 
-  ami                   = data.aws_ami.rhel8.image_id
+  ami                   = data.aws_ami.server.image_id
   ssh_authorized_keys   = [tls_private_key.ssh.public_key_openssh]
-  instance_type         = "t3.medium"
+  instance_type         = "m5a.large"
   controlplane_internal = false # Note this defaults to best practice of true, but is explicitly set to public for demo purposes
   servers               = 1
 
   # Enable AWS Cloud Controller Manager
   enable_ccm        = true
   enable_autoscaler = true
 
-  rke2_config = yamlencode({ "node-label" : ["name=server", "os=rhel8"] })
+  rke2_config = yamlencode({ "node-label" : ["name=server", "os=rhel"] })
 
-  rke2_channel = "v1.27"
+  rke2_channel = "v1.33"
 }
 
 #
@@ -121,21 +123,21 @@ module "agents" {
   vpc_id  = module.vpc.vpc_id
   subnets = module.vpc.public_subnets # Note: Public subnets used for demo purposes, this is not recommended in production
 
-  ami                 = data.aws_ami.rhel8.image_id
+  ami                 = data.aws_ami.server.image_id
   ssh_authorized_keys = [tls_private_key.ssh.public_key_openssh]
   spot                = true
-  asg                 = { min : 1, max : 10, desired : 2 }
-  instance_type       = "t3.large"
+  asg                 = { min : 2, max : 2, desired : 2 }
+  instance_type       = "m5a.xlarge"
 
   # Enable AWS Cloud Controller Manager and Cluster Autoscaler
   enable_ccm        = true
   enable_autoscaler = true
 
-  rke2_config = yamlencode({ "node-label" : ["name=generic", "os=rhel8"] })
+  rke2_config = yamlencode({ "node-label" : ["name=generic", "os=rhel"] })
 
   cluster_data = module.rke2.cluster_data
 
-  rke2_channel = "v1.27"
+  rke2_channel = "v1.33"
 }
 
 # For demonstration only, lock down ssh access in production

modules/agent-nodepool/README.md

@@ -36,11 +36,11 @@
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_ami"></a> [ami](#input\_ami) | Node pool ami | `string` | `""` | no |
-| <a name="input_asg"></a> [asg](#input\_asg) | Node pool AutoScalingGroup scaling definition | <pre>object({<br>    min                  = number<br>    max                  = number<br>    desired              = number<br>    suspended_processes  = optional(list(string))<br>    termination_policies = optional(list(string))<br>  })</pre> | <pre>{<br>  "desired": 1,<br>  "max": 10,<br>  "min": 1,<br>  "suspended_processes": [],<br>  "termination_policies": [<br>    "Default"<br>  ]<br>}</pre> | no |
+| <a name="input_asg"></a> [asg](#input\_asg) | Node pool AutoScalingGroup scaling definition | <pre>object({<br/>    min                  = number<br/>    max                  = number<br/>    desired              = number<br/>    suspended_processes  = optional(list(string))<br/>    termination_policies = optional(list(string))<br/>  })</pre> | <pre>{<br/>  "desired": 1,<br/>  "max": 10,<br/>  "min": 1,<br/>  "suspended_processes": [],<br/>  "termination_policies": [<br/>    "Default"<br/>  ]<br/>}</pre> | no |
 | <a name="input_awscli_url"></a> [awscli\_url](#input\_awscli\_url) | URL for awscli zip file | `string` | `"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"` | no |
-| <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | Node pool block device mapping configuration | `map(string)` | <pre>{<br>  "size": 30,<br>  "type": "gp2"<br>}</pre> | no |
+| <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | Node pool block device mapping configuration | `map(string)` | <pre>{<br/>  "size": "30"<br/>}</pre> | no |
 | <a name="input_ccm_external"></a> [ccm\_external](#input\_ccm\_external) | Set kubelet arg 'cloud-provider-name' value to 'external'.  Requires manual install of CCM. | `bool` | `false` | no |
-| <a name="input_cluster_data"></a> [cluster\_data](#input\_cluster\_data) | Required data relevant to joining an existing rke2 cluster, sourced from main rke2 module, do NOT modify | <pre>object({<br>    name       = string<br>    server_url = string<br>    cluster_sg = string<br>    token = object({<br>      bucket          = string<br>      bucket_arn      = string<br>      object          = string<br>      policy_document = string<br>    })<br>  })</pre> | n/a | yes |
+| <a name="input_cluster_data"></a> [cluster\_data](#input\_cluster\_data) | Required data relevant to joining an existing rke2 cluster, sourced from main rke2 module, do NOT modify | <pre>object({<br/>    name       = string<br/>    server_url = string<br/>    cluster_sg = string<br/>    token = object({<br/>      bucket          = string<br/>      bucket_arn      = string<br/>      object          = string<br/>      policy_document = string<br/>    })<br/>  })</pre> | n/a | yes |
 | <a name="input_download"></a> [download](#input\_download) | Toggle best effort download of rke2 dependencies (rke2 and aws cli), if disabled, dependencies are assumed to exist in $PATH | `bool` | `true` | no |
 | <a name="input_enable_autoscaler"></a> [enable\_autoscaler](#input\_enable\_autoscaler) | Toggle configure the nodepool for cluster autoscaler, this will ensure the appropriate IAM policies are present, you are still responsible for ensuring cluster autoscaler is installed | `bool` | `false` | no |
 | <a name="input_enable_ccm"></a> [enable\_ccm](#input\_enable\_ccm) | Toggle enabling the cluster as aws aware, this will ensure the appropriate IAM policies are present | `bool` | `false` | no |
@@ -50,7 +50,7 @@
 | <a name="input_iam_instance_profile"></a> [iam\_instance\_profile](#input\_iam\_instance\_profile) | Node pool IAM Instance Profile, created if left blank (default behavior) | `string` | `""` | no |
 | <a name="input_iam_permissions_boundary"></a> [iam\_permissions\_boundary](#input\_iam\_permissions\_boundary) | If provided, the IAM role created for the nodepool will be created with this permissions boundary attached. | `string` | `null` | no |
 | <a name="input_instance_type"></a> [instance\_type](#input\_instance\_type) | Node pool instance type | `string` | `"t3.medium"` | no |
-| <a name="input_metadata_options"></a> [metadata\_options](#input\_metadata\_options) | Instance Metadata Options | `map(any)` | <pre>{<br>  "http_endpoint": "enabled",<br>  "http_put_response_hop_limit": 2,<br>  "http_tokens": "required",<br>  "instance_metadata_tags": "disabled"<br>}</pre> | no |
+| <a name="input_metadata_options"></a> [metadata\_options](#input\_metadata\_options) | Instance Metadata Options | `map(any)` | <pre>{<br/>  "http_endpoint": "enabled",<br/>  "http_put_response_hop_limit": 2,<br/>  "http_tokens": "required",<br/>  "instance_metadata_tags": "disabled"<br/>}</pre> | no |
 | <a name="input_name"></a> [name](#input\_name) | Nodepool name | `string` | n/a | yes |
 | <a name="input_post_userdata"></a> [post\_userdata](#input\_post\_userdata) | Custom userdata to run immediately after rke2 node attempts to join cluster | `string` | `""` | no |
 | <a name="input_pre_userdata"></a> [pre\_userdata](#input\_pre\_userdata) | Custom userdata to run immediately before rke2 node attempts to join cluster, after required rke2, dependencies are installed | `string` | `""` | no |