You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Karpenter add-on is based on the [Karpenter](https://github.com/aws/karpenter) open source node provisioning project. It provides a more efficient and cost-effective way to manage workloads by launching just the right compute resources to handle a cluster's application.
3
+
Karpenter add-on is based on the [Karpenter](https://github.com/kubernetes-sigs/karpenter) open source node provisioning project. For this add-on, it will utilize the [AWS provider](https://github.com/aws/karpenter-provider-aws), to ensure a more efficient and cost-effective way to manage workloads by launching just the right compute resources to handle a cluster's application on your EKS cluster.
4
4
5
5
Karpenter works by:
6
6
@@ -16,66 +16,63 @@ Karpenter works by:
16
16
17
17
2. (If using Spot), EC2 Spot Service Linked Role should be created. See [here](https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html) for more details.
18
18
19
+
3.[Amazon EKS cluster with supported Kubernetes version](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html). Karpenter provides minimum supported Karpenter versions for each Kubernetes version in form of a matrix [here](https://karpenter.sh/docs/upgrading/compatibility/#compatibility-matrix).
The add-on automatically sets the following Helm Chart [values](https://github.com/aws/karpenter/tree/main/charts/karpenter#values), and it is **highly recommended** not to pass these values in (as it will result in errors):
75
-
- settings.aws.defaultInstanceProfile
76
-
- settings.aws.clusterEndpoint
77
-
- settings.aws.clusterName
78
-
- settings.aws.interruptionQueueName (if interruption handling is enabled)
73
+
The add-on automatically sets Helm Chart [values](https://github.com/aws/karpenter-provider-aws/tree/main/charts/karpenter#values), and it is **recommended** not to pass custom values for the following:
74
+
- settings.clusterName
75
+
- settings.interruptionQueue (if interruption handling is enabled)
1. Creates Karpenter Node Role, Karpenter Instance Profile, and Karpenter Controller Policy (Please see Karpenter documentation [here](https://karpenter.sh/docs/getting-started/) for more details on what is required and why).
95
92
2. Creates `karpenter` namespace.
96
93
3. Creates Kubernetes Service Account, and associate AWS IAM Role with Karpenter Controller Policy attached using [IRSA](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html).
97
-
4. Deploys Karpenter helm chart in the `karpenter` namespace, configuring cluster name and cluster endpoint on the controller by default.
98
-
5. (Optionally) provisions a default Karpenter Provisioner and AWSNodeTemplate CRD based on user-provided parameters such as [spec.requirements](https://karpenter.sh/docs/concepts/nodepools/#spectemplatespecrequirements), [AMI type](https://karpenter.sh/docs/concepts/nodeclasses/#specamifamily),[weight](https://karpenter.sh/docs/concepts/provisioners/#specweight), [Subnet Selector](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnetselectorterms), [Security Group Selector](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms), [Tags](https://karpenter.sh/docs/concepts/nodeclasses/#spectags) and [BlockDeviceMappings](https://karpenter.sh/docs/concepts/node-templates/#specblockdevicemappings). If created, the provisioner will discover the EKS VPC subnets and security groups to launch the nodes with.
94
+
4. Deploys Karpenter helm chart in the `karpenter` namespace, configuring the cluster name, endpoint, Instance Profile, and others necessary for functional addon.
95
+
5. If the user provides `nodePoolSpec` (and `ec2NodeClassSpec`), the addon will provisions a default Karpenter NodePool and EC2NodeClass CRDs. `nodePoolSpec` requires [requirements](https://karpenter.sh/docs/concepts/nodepools/#spectemplatespecrequirements) while `ec2NodeClassSpec` requires subnets and security groups. Based on what version of Karpenter you provide, you will need either `subnetSelector` and `securityGroupSelector` (for versions v0.31.x or down), or `subnetSelectorTerms` and `securityGroupSelectorTerms` (for versions v0.32.x and up).
96
+
6. As mentioned above, the CRDs installed will be different from v0.32.0, since Karpenter as a project graduated to beta in October 2023. This meant significant API changes, going from alpha to beta. The addon has reflected those changes and will deploy NodePool and EC2NodeClass for v1beta1 CRDs, versus Provisioner and AWSNodeTemplate for v1alpha5. You can read more about the changes in this [blog](https://aws.amazon.com/blogs/containers/karpenter-graduates-to-beta/).
99
97
100
-
**NOTE:**
101
-
1. The default provisioner is created only if both the subnet tags and the security group tags are provided.
102
-
2. Provisioner spec requirement fields are not necessary, as karpenter will dynamically choose (i.e. leaving instance-type blank will let karpenter choose appropriate sizing).
103
-
3. Consolidation, which is a flag that enables , is supported on versions 0.15.0 and later. It is also mutually exclusive with `ttlSecondsAfterEmpty`, so if you provide both properties, the addon will throw an error.
104
-
4. Weight, which is a property to prioritize provisioners based on weight, is supported on versions 0.16.0 and later. Addon will throw an error if weight is provided for earlier versions.
105
-
5. Interruption Handling, which is a native way to handle interruption due to involuntary interruption events, is supported on versions 0.19.0 and later. For interruption handling in the earlier versions, Karpenter supports using AWS Node Interruption Handler (which you will need to add as an add-on and ***must be in add-on array after the Karpenter add-on*** for it to work.
106
-
6. Karpenter allows overrides of the default "Name" tag but does not allow overrides to restricted domain (such as "karpenter.sh", "karpenter.k8s.aws", and "kubernetes.io/cluster").
98
+
***NOTE: EKS Blueprints npm v1.14 and above introduces breaking changes to the addon. Please see [Upgrade Path](#upgrade-path) for more details.***
107
99
108
100
## Using Karpenter
109
101
110
-
To use Karpenter, you need to provision a Karpenter [provisioner CRD](https://karpenter.sh/docs/concepts/provisioners/). A single provisioner is capable of handling many different pod shapes.
102
+
To use Karpenter, you need to provision a Karpenter [NodePool](https://karpenter.sh/docs/concepts/nodepools/) and [EC2NodeClass](https://karpenter.sh/docs/concepts/nodeclasses/). NodePool sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes. EC2NodeClass, once associated with a NodePool, will then provision those nodes (in the form of EC2 instances) based on the AWS specific settings. Multiple NodePools may point to the same EC2NodeClass.
111
103
112
104
This can be done in 2 ways:
113
105
114
-
1. Provide the properties as show in [Usage](#usage). If subnet tags and security group tags are not provided at deploy time, the add-on will be installed without a Provisioner.
106
+
1. Provide the properties as show in [Usage](#usage). If the NodePoolSpec is not provided, the addon will not deploy a NodePool or EC2NodeClass.
115
107
116
-
2. Use `kubectl` to apply a sample provisioner manifest:
108
+
2. Use `kubectl` to apply a sample NodePool and EC2NodeClass:
117
109
```bash
118
110
cat <<EOF | kubectl apply -f -
119
-
apiVersion: karpenter.sh/v1alpha5
120
-
kind: Provisioner
111
+
apiVersion: karpenter.sh/v1beta1
112
+
kind: NodePool
113
+
metadata:
114
+
name: default
115
+
spec:
116
+
template:
117
+
spec:
118
+
nodeClassRef:
119
+
name: default
120
+
---
121
+
apiVersion: karpenter.k8s.aws/v1beta1
122
+
kind: EC2NodeClass
121
123
metadata:
122
124
name: default
123
125
spec:
124
-
requirements:
125
-
- key: "node.kubernetes.io/instance-type"
126
-
operator: In
127
-
values: ["m5.2xlarge"]
128
-
- key: "topology.kubernetes.io/zone"
129
-
operator: In
130
-
values: ["us-east-1c"]
131
-
- key: "kubernetes.io/arch"
132
-
operator: In
133
-
values: ["arm64", "amd64"]
134
-
- key: "karpenter.sh/capacity-type"
135
-
operator: In
136
-
values: ["spot", "on-demand"]
137
-
provider:
138
-
instanceProfile: <<Name of your Instance Profile>>
# Select on any security group that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag
133
+
# AND the "environment: test" tag OR any security group with the "my-security-group" name
134
+
# OR any security group with ID "sg-063d7acfb4b06c82c"
135
+
- tags:
136
+
karpenter.sh/discovery: "${CLUSTER_NAME}"
137
+
138
+
role: "KarpenterNodeRole-${CLUSTER_NAME}"
139
+
140
+
userData: |
141
+
echo "Hello world"
142
+
143
+
tags:
144
+
team: team-a
145
+
app: team-a-app
146
+
147
+
metadataOptions:
148
+
httpEndpoint: enabled
149
+
httpProtocolIPv6: disabled
150
+
httpPutResponseHopLimit: 2
151
+
httpTokens: required
152
+
153
+
blockDeviceMappings:
154
+
- deviceName: /dev/xvda
155
+
ebs:
156
+
volumeSize: 100Gi
157
+
volumeType: gp3
158
+
iops: 10000
159
+
encrypted: true
160
+
kmsKeyID: "1234abcd-12ab-34cd-56ef-1234567890ab"
161
+
deleteOnTermination: true
162
+
throughput: 125
163
+
snapshotID: snap-0123456789
164
+
165
+
detailedMonitoring: true
166
+
144
167
EOF
145
168
```
146
169
147
-
If you choose to create a provisioner manually, you **MUST** provide the tags that match the subnet and the security group that you want to use.
170
+
If you choose to create NodePool and EC2NodeClass manually, you **MUST** provide the tags that match the subnet and the security group from the Blueprints EKS cluster that you plan to use.
148
171
149
172
## Testing with a sample deployment
150
173
@@ -213,35 +236,15 @@ Karpenter, starting from the OCI registry versions, will untar the files under `
213
236
214
237
## Upgrade Path
215
238
216
-
1. Using an older version of the Karptner add-on, you may notice the difference in the "provisionerSpecs" property:
The property is changed to align with the naming convention of the provisioner, and to allow multiple operators (In vs NotIn). The values correspond similarly between the two, with type change being the only difference.
239
+
The addon introduces breaking changes for Blueprints npm version v0.14 and later. Here are the details:
239
240
240
-
2. Certain upgrades require reapplying the CRDs since Helm does not maintain the lifecycle of CRDs. Please see the [official documentations](hhttps://karpenter.sh/docs/upgrading/upgrade-guide/) for details.
241
+
- EKS Blueprints will only support minimum Karpenter version that matches the supporter EKS Kubernetes version. Please see the compatibility matrix [here](https://karpenter.sh/docs/upgrading/compatibility/). If you provide incompatible version (i.e. providing version 0.27.x for EKS version 1.27), you will see warnings in the logs but will proceed deployment. You will run into compatibility issues.
242
+
- The add-on will no longer support any versions below v0.21.0
243
+
- User provided properties have been refactored to better reflect the parameters of the various Karpenter resources (i.e. NodePool, EC2NodeClass)
244
+
- For NodePool and EC2NodeClass, the parameters will apply to either the v1alpha5 CRDs ( provisioner, AWSNodeTemplate, for Karpenter versions v0.31.x or earlier) or v1beta1 CRDs (NodePool, EC2NodeClass, for Karpenter versions v0.32.x and later). **If you provide non-matching parameters, i.e. providing `consolidation` instead of `disruption` for Karpenter version v0.33.1, you will see an error with stack failing to provision.** Please consult the [upgrade guide](https://karpenter.sh/docs/upgrading/upgrade-guide/) to see the changes for various versions.
241
245
242
-
3. Starting with v0.17.0, Karpenter's Helm chart package is stored in OCI (Open Container Initiative) registry. With this change, [charts.karpenter.sh](https://charts.karpenter.sh/) is no longer updated to preserve older versions. You have to adjust for the following:
246
+
If you are upgrading from earlier version of Blueprints and need to add the Karpenter addon, please ensure the following:
243
247
244
-
* The full URL needs to be present (including 'oci://').
245
-
* You need to append a `v` to the version number (i.e. v0.17.0, not 0.17.0)
248
+
1. You are using the minimum Karpenter version supported by the Kubernetes version of your blueprint cluster. Not doing so will cause incompatibility issues.
246
249
247
-
4. Starting with v0.22.0, Karpenter will no longer work on Kubernetes version prior to 1.21. Either upgrade your Kubernetes to 1.21 or later version and apply Karpenter, or use prior Karpenter versions.
250
+
2. Starting v0.32.0, Karpenter introduces the new beta APIs (v1beta1), and therefore the addon will make v1alpha5 CRDs obsolete. Ensure that you are providing the corresponding, matching parameters.
0 commit comments