Skip to content

feat: add eks auto mode #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from
Closed
Show file tree
Hide file tree
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,7 @@ Available targets:
| <a name="input_cloudwatch_log_group_class"></a> [cloudwatch\_log\_group\_class](#input\_cloudwatch\_log\_group\_class) | Specified the log class of the log group. Possible values are: `STANDARD` or `INFREQUENT_ACCESS` | `string` | `null` | no |
| <a name="input_cloudwatch_log_group_kms_key_id"></a> [cloudwatch\_log\_group\_kms\_key\_id](#input\_cloudwatch\_log\_group\_kms\_key\_id) | If provided, the KMS Key ID to use to encrypt AWS CloudWatch logs | `string` | `null` | no |
| <a name="input_cluster_attributes"></a> [cluster\_attributes](#input\_cluster\_attributes) | Override label module default cluster attributes | `list(string)` | <pre>[<br/> "cluster"<br/>]</pre> | no |
| <a name="input_cluster_compute_config"></a> [cluster\_compute\_config](#input\_cluster\_compute\_config) | Configuration block with compute configuration for EKS Auto Mode<br/><br/>enabled: Request to enable or disable the compute capability on your EKS Auto Mode cluster. If the compute capability is enabled, EKS Auto Mode will create and delete EC2 Managed Instances in your Amazon Web Services account.<br/>node\_pools: Optional configuration for node pools that defines the compute resources for your EKS Auto Mode cluster. Valid options are general-purpose and system.<br/>node\_role\_arn: Optional ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS Auto Mode cluster. | <pre>object({<br/> enabled = optional(bool, false)<br/> node_pools = optional(list(string), [])<br/> node_role_arn = optional(string, null)<br/> })</pre> | `{}` | no |
| <a name="input_cluster_depends_on"></a> [cluster\_depends\_on](#input\_cluster\_depends\_on) | If provided, the EKS will depend on this object, and therefore not be created until this object is finalized.<br/>This is useful if you want to ensure that the cluster is not created before some other condition is met, e.g. VPNs into the subnet are created. | `any` | `null` | no |
| <a name="input_cluster_encryption_config_enabled"></a> [cluster\_encryption\_config\_enabled](#input\_cluster\_encryption\_config\_enabled) | Set to `true` to enable Cluster Encryption Configuration | `bool` | `true` | no |
| <a name="input_cluster_encryption_config_kms_key_deletion_window_in_days"></a> [cluster\_encryption\_config\_kms\_key\_deletion\_window\_in\_days](#input\_cluster\_encryption\_config\_kms\_key\_deletion\_window\_in\_days) | Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction | `number` | `10` | no |
Expand Down
1 change: 1 addition & 0 deletions docs/terraform.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@
| <a name="input_cloudwatch_log_group_class"></a> [cloudwatch\_log\_group\_class](#input\_cloudwatch\_log\_group\_class) | Specified the log class of the log group. Possible values are: `STANDARD` or `INFREQUENT_ACCESS` | `string` | `null` | no |
| <a name="input_cloudwatch_log_group_kms_key_id"></a> [cloudwatch\_log\_group\_kms\_key\_id](#input\_cloudwatch\_log\_group\_kms\_key\_id) | If provided, the KMS Key ID to use to encrypt AWS CloudWatch logs | `string` | `null` | no |
| <a name="input_cluster_attributes"></a> [cluster\_attributes](#input\_cluster\_attributes) | Override label module default cluster attributes | `list(string)` | <pre>[<br/> "cluster"<br/>]</pre> | no |
| <a name="input_cluster_compute_config"></a> [cluster\_compute\_config](#input\_cluster\_compute\_config) | Configuration block with compute configuration for EKS Auto Mode<br/><br/>enabled: Request to enable or disable the compute capability on your EKS Auto Mode cluster. If the compute capability is enabled, EKS Auto Mode will create and delete EC2 Managed Instances in your Amazon Web Services account.<br/>node\_pools: Optional configuration for node pools that defines the compute resources for your EKS Auto Mode cluster. Valid options are general-purpose and system.<br/>node\_role\_arn: Optional ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS Auto Mode cluster. | <pre>object({<br/> enabled = optional(bool, false)<br/> node_pools = optional(list(string), [])<br/> node_role_arn = optional(string, null)<br/> })</pre> | `{}` | no |
| <a name="input_cluster_depends_on"></a> [cluster\_depends\_on](#input\_cluster\_depends\_on) | If provided, the EKS will depend on this object, and therefore not be created until this object is finalized.<br/>This is useful if you want to ensure that the cluster is not created before some other condition is met, e.g. VPNs into the subnet are created. | `any` | `null` | no |
| <a name="input_cluster_encryption_config_enabled"></a> [cluster\_encryption\_config\_enabled](#input\_cluster\_encryption\_config\_enabled) | Set to `true` to enable Cluster Encryption Configuration | `bool` | `true` | no |
| <a name="input_cluster_encryption_config_kms_key_deletion_window_in_days"></a> [cluster\_encryption\_config\_kms\_key\_deletion\_window\_in\_days](#input\_cluster\_encryption\_config\_kms\_key\_deletion\_window\_in\_days) | Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction | `number` | `10` | no |
Expand Down
5 changes: 5 additions & 0 deletions examples/complete/fixtures.us-east-2.tfvars
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,8 @@ upgrade_policy = {
zonal_shift_config = {
enabled = true
}

cluster_compute_config = {
enabled = false
}
Comment on lines +61 to +63
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but for something this complex and impactful, in addition to the specific changes I called out elsewhere, I want to see a separate test of auto mode in the automated testing. It can run in parallel.

Because of a quirk in Terratest, vars set in a test do not override vars set in a varfile set in the test, so my plan would be:

  • Put a full configuration of the enabled cluster_compute_config in the varfile.
  • Add a var to the example: cluster_auto_mode_enabled which defaults to false.
  • Update the main.tf to honor that var. This would include, if I understand correctly, not creating the eks_node_group when in auto mode (enabled = !var.cluster_auto_mode_enabled).
  • Clone TestExamplesComplete to create TestExamplesAutoMode, adding "cluster_auto_mode_enabled": true to the test vars. Adjust the tests as appropriate to ensure we have a functioning cluster in auto mode.


1 change: 1 addition & 0 deletions examples/complete/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ module "eks_cluster" {
bootstrap_self_managed_addons_enabled = var.bootstrap_self_managed_addons_enabled
upgrade_policy = var.upgrade_policy
zonal_shift_config = var.zonal_shift_config
cluster_compute_config = var.cluster_compute_config

access_entry_map = local.access_entry_map
access_config = {
Expand Down
16 changes: 16 additions & 0 deletions examples/complete/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,22 @@ variable "zonal_shift_config" {
default = null
}

variable "cluster_compute_config" {
type = object({
enabled = optional(bool, false)
node_pools = optional(list(string), [])
node_role_arn = optional(string, null)
})
description = <<-EOT
Configuration block with compute configuration for EKS Auto Mode

enabled: Request to enable or disable the compute capability on your EKS Auto Mode cluster. If the compute capability is enabled, EKS Auto Mode will create and delete EC2 Managed Instances in your Amazon Web Services account.
node_pools: Optional configuration for node pools that defines the compute resources for your EKS Auto Mode cluster. Valid options are general-purpose and system.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if left undefined?

node_role_arn: Optional ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS Auto Mode cluster.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any gotchas with the default IAM role EKS will use, e.g edge cases? If so, then we should probably have an option to create and use the role we create when we use the old/existing node pool. In fact, that should probably be an option in any case.

EOT
default = {}
}

variable "private_ipv6_enabled" {
type = bool
default = false
Expand Down
53 changes: 46 additions & 7 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ locals {
}

cloudwatch_log_group_name = "/aws/eks/${module.label.id}/cluster"

auto_mode_enabled = try(var.cluster_compute_config.enabled, false)
}

module "label" {
Expand Down Expand Up @@ -56,19 +58,31 @@ resource "aws_kms_alias" "cluster" {
resource "aws_eks_cluster" "default" {
#bridgecrew:skip=BC_AWS_KUBERNETES_1:Allow permissive security group for public access, difficult to restrict without a VPN
#bridgecrew:skip=BC_AWS_KUBERNETES_4:Let user decide on control plane logging, not necessary in non-production environments
count = local.enabled ? 1 : 0
name = module.label.id
tags = module.label.tags
role_arn = local.eks_service_role_arn
version = var.kubernetes_version
enabled_cluster_log_types = var.enabled_cluster_log_types
bootstrap_self_managed_addons = var.bootstrap_self_managed_addons_enabled
count = local.enabled ? 1 : 0
name = module.label.id
tags = module.label.tags
role_arn = local.eks_service_role_arn
version = var.kubernetes_version
enabled_cluster_log_types = var.enabled_cluster_log_types
# Enabling EKS Auto Mode also requires that bootstrap_self_managed_addons is set to false
bootstrap_self_managed_addons = local.auto_mode_enabled ? false : var.bootstrap_self_managed_addons_enabled
Comment on lines +67 to +68
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I consider it a bad idea to silently change input configuration like this.

Because this is a configuration conflict, and may change in the future, my first preference is to allow AWS to throw an error message rather than try to fix it, because then if AWS relaxes this condition, we don't need to make any changes.

An acceptable alternative is to add this check as a precondition and report an error.


access_config {
authentication_mode = var.access_config.authentication_mode
bootstrap_cluster_creator_admin_permissions = var.access_config.bootstrap_cluster_creator_admin_permissions
}

# EKS Auto Mode
dynamic "compute_config" {
for_each = local.auto_mode_enabled && (var.cluster_compute_config) > 0 ? [var.cluster_compute_config] : []

content {
enabled = local.auto_mode_enabled
node_pools = local.auto_mode_enabled ? try(compute_config.value.node_pools, []) : null
node_role_arn = local.auto_mode_enabled && length(try(compute_config.value.node_pools, [])) > 0 ? try(compute_config.value.node_role_arn, aws_iam_role.default[0].arn, null) : null
}
}

lifecycle {
# bootstrap_cluster_creator_admin_permissions is documented as only applying
# to the initial creation of the cluster, and being unreliable afterward,
Expand Down Expand Up @@ -98,18 +112,43 @@ resource "aws_eks_cluster" "default" {

dynamic "kubernetes_network_config" {
for_each = local.use_ipv6 ? [] : compact([var.service_ipv4_cidr])

content {
service_ipv4_cidr = kubernetes_network_config.value
}
}

dynamic "kubernetes_network_config" {
for_each = local.use_ipv6 ? [true] : []

content {
ip_family = "ipv6"
}
}

dynamic "kubernetes_network_config" {
for_each = local.auto_mode_enabled ? [1] : []

content {
dynamic "elastic_load_balancing" {
for_each = local.auto_mode_enabled ? [1] : []
content {
enabled = local.auto_mode_enabled
}
Comment on lines +133 to +137
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really a requirement that elastic load balancing be enabled for auto mode? If so:

Suggested change
dynamic "elastic_load_balancing" {
for_each = local.auto_mode_enabled ? [1] : []
content {
enabled = local.auto_mode_enabled
}
elastic_load_balancing {
enabled = true
}

If it not a requirement, then it should be an option in the auto mode configuration object.

}
}
}

dynamic "storage_config" {
for_each = local.auto_mode_enabled ? [1] : []

content {
block_storage {
enabled = local.auto_mode_enabled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, if required, then just use true to emphasize the fact. If not required, make it a separate input option.

Suggested change
enabled = local.auto_mode_enabled
enabled = true

}
}
}

dynamic "upgrade_policy" {
for_each = var.upgrade_policy != null ? [var.upgrade_policy] : []
content {
Expand Down
18 changes: 16 additions & 2 deletions variables.tf
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
# tflint-ignore: terraform_unused_declarations
variable "region" {

type = string
description = "OBSOLETE (not needed): AWS Region"
default = null
Expand Down Expand Up @@ -29,6 +28,22 @@ variable "cluster_depends_on" {
default = null
}

variable "cluster_compute_config" {
type = object({
enabled = optional(bool, false)
node_pools = optional(list(string), [])
node_role_arn = optional(string, null)
})
description = <<-EOT
Configuration block with compute configuration for EKS Auto Mode

enabled: Request to enable or disable the compute capability on your EKS Auto Mode cluster. If the compute capability is enabled, EKS Auto Mode will create and delete EC2 Managed Instances in your Amazon Web Services account.
node_pools: Optional configuration for node pools that defines the compute resources for your EKS Auto Mode cluster. Valid options are general-purpose and system.
node_role_arn: Optional ARN of the IAM Role EKS will assign to EC2 Managed Instances in your EKS Auto Mode cluster.
EOT
default = {}
}

variable "create_eks_service_role" {
type = bool
description = "Set `false` to use existing `eks_cluster_service_role_arn` instead of creating one"
Expand All @@ -45,7 +60,6 @@ variable "eks_cluster_service_role_arn" {
default = null
}


variable "kubernetes_version" {
type = string
description = "Desired Kubernetes master version. If you do not specify a value, the latest available version is used"
Expand Down