Skip to content

Commit 62cb956

Browse files
woehrl01woehrl01
authored
Merge branch 'main' into multi-dns
2 parents 1197759 + 69ed8b9 commit 62cb956

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+2001
-1701
lines changed

‎ADOPTERS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,5 +59,6 @@ If you are open to others contacting you about your use of Karpenter on Slack, a
5959
| Whoosh | Using Karpenter to scale the EKS clusters for many purposes | `@vainkop` | [Whoosh](https://whoosh.bike) |
6060
| Next Insurance | Using Karpenter to manage the nodes in all our EKS clusters, including dev and prod, on demand and spots | `@moshebs` | [Homepage](https://www.nextinsurance.com)|
6161
| Grover Group GmbH | We use Karpenter for efficient and cost effective scaling of our nodes in all of our EKS clusters | `@suraj2410` | [Homepage](https://www.grover.com/de-en) & [Engineering Techblog](https://engineering.grover.com)|
62+
| Legit Security | We run Karpenter across all our EKS clusters to ensure efficient and cost-effective scaling across our infrastructure | `@Tal Balash`, `@Matan Ryngler` | [Homepage](https://www.legitsecurity.com)|
6263
| Logz.io | Using Karpenter in all of our EKS clusters for efficient and cost effective scaling of all our K8s workloads | `@pincher95`, `@Samplify` | [Homepage](https://logz.io/)|
6364
| X3M ads | We have been using Karpenter for (almost) all our workloads since 2023 | `@mreparaz`, `@fmansilla`, `@mrmartinez95` | [Homepage](https://x3mads.com) |

‎charts/karpenter-crd/templates/karpenter.sh_nodeclaims.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,10 @@ spec:
3838
- jsonPath: .metadata.creationTimestamp
3939
name: Age
4040
type: date
41+
- jsonPath: .status.imageID
42+
name: ImageID
43+
priority: 1
44+
type: string
4145
- jsonPath: .status.providerID
4246
name: ID
4347
priority: 1

‎cmd/controller/main.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121

2222
"sigs.k8s.io/karpenter/pkg/cloudprovider/metrics"
2323
corecontrollers "sigs.k8s.io/karpenter/pkg/controllers"
24+
"sigs.k8s.io/karpenter/pkg/controllers/state"
2425
coreoperator "sigs.k8s.io/karpenter/pkg/operator"
2526
)
2627

@@ -36,6 +37,7 @@ func main() {
3637
op.SecurityGroupProvider,
3738
)
3839
cloudProvider := metrics.Decorate(awsCloudProvider)
40+
clusterState := state.NewCluster(op.Clock, op.GetClient(), cloudProvider)
3941

4042
op.
4143
WithControllers(ctx, corecontrollers.NewControllers(
@@ -45,6 +47,7 @@ func main() {
4547
op.GetClient(),
4648
op.EventRecorder,
4749
cloudProvider,
50+
clusterState,
4851
)...).
4952
WithControllers(ctx, controllers.NewControllers(
5053
ctx,

‎designs/interruption-handling.md

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,17 @@ There are two ways in-which Spot interruption notifications and Rebalance Recomm
2929
EC2 IMDS is an HTTP API that can only be locally accessed from an EC2 instance.
3030

3131
```
32-
`curl 169.254.169.254/latest/meta-data/spot/instance-action
32+
# Termination Check
33+
curl 169.254.169.254/latest/meta-data/spot/instance-action
3334
{
3435
"action": "terminate",
3536
"time": "2022-07-11T17:11:44Z"
3637
}
3738
38-
curl 169.254.169.254``/``latest``/``meta``-``data``/``events``/``recommendations``/``rebalance`
39-
`{`
40-
` ``"noticeTime"``:`` ``"2022-07-16T19:18:24Z"`
39+
# Rebalance Check
40+
curl 169.254.169.254/latest/meta-data/events/recommendations/rebalance
41+
{
42+
"noticeTime": "2022-07-16T19:18:24Z"
4143
}
4244
4345
```
@@ -47,19 +49,19 @@ curl 169.254.169.254``/``latest``/``meta``-``data``/``events``/``recommendations
4749
EventBridge is an Event Bus service within AWS that allows users to set rules on events to capture and then target destinations for those events. Relevant targets for Spot interruption notifications include SQS, Lambda, and EC2-Terminate-Instance.
4850

4951
```
50-
`# Example spot interruption notification EventBridge rule`
51-
`$ aws events put``-``rule \`
52-
` ``--``name ``MyK8sSpotTermRule`` \`
53-
` ``--``event``-``pattern ``"{\"source\": [\"aws.ec2\"],\"detail-type\": [\"EC2 Spot Instance Interruption\"]}"`
54-
55-
`# Example rebalance recommendation EventBridge rule``
56-
$ aws events put-rule \
57-
--name MyK8sRebalanceRule \
58-
--event-pattern "{\"source\": [\"aws.ec2\"],\"detail-type\": [\"EC2 Instance Rebalance Recommendation\"]}"
59-
`` `
60-
`# Example targeting an SQS queue`
61-
`$ aws events put``-``targets ``--``rule ``MyK8sSpotTermRule`` \`
62-
` ``--``targets ``"Id"``=``"1"``,``"Arn"``=``"arn:aws:sqs:us-east-1:123456789012:MyK8sTermQueue"`` `
52+
# Example spot interruption notification EventBridge rule
53+
aws events put-rule \
54+
--name MyK8sSpotTermRule \
55+
--event-pattern "{\"source\": [\"aws.ec2\"],\"detail-type\": [\"EC2 Spot Instance Interruption\"]}"
56+
57+
# Example rebalance recommendation EventBridge rule
58+
aws events put-rule \
59+
--name MyK8sRebalanceRule \
60+
--event-pattern "{\"source\": [\"aws.ec2\"],\"detail-type\": [\"EC2 Instance Rebalance Recommendation\"]}"
61+
62+
# Example targeting an SQS queue
63+
aws events put-targets --rule MyK8sSpotTermRule \
64+
--targets "Id=1,Arn=arn:aws:sqs:us-east-1:123456789012:MyK8sTermQueue"
6365
```
6466

6567

@@ -113,17 +115,17 @@ SQS exposes a VPC Endpoint which will fulfill the isolated VPC use-case.
113115
Dynamically creating the SQS infrastructure and EventBridge rules means that Karpenter’s IAM role would need permissions to SQS and EventBridge:
114116

115117
```
116-
`"sqs:GetQueueUrl",`
117-
`"sqs:ListQueues"``,`
118-
`"sqs:ReceiveMessage"``,`
119-
`"sqs:CreateQueue"``,`
120-
`"sqs:DeleteMessage"``,`
121-
`"events:ListRules",`
122-
"`events:DescribeRule`",
123-
"events:PutRule",
118+
"sqs:GetQueueUrl",
119+
"sqs:ListQueues",
120+
"sqs:ReceiveMessage",
121+
"sqs:CreateQueue",
122+
"sqs:DeleteMessage",
123+
"events:ListRules",
124+
"events:DescribeRule",
125+
"events:PutRule",
124126
"events:PutTargets",
125-
"`events:DeleteRule`",
126-
`"events:RemoveTargets"`
127+
"events:DeleteRule",
128+
"events:RemoveTargets"
127129
```
128130

129131
The policy can be setup with a predefined name based on the cluster name. For example, `karpenter-events-${CLUSTER_NAME}` which would allow for a more constrained resource policy.
@@ -144,7 +146,7 @@ The simplest option is to include [NTH IMDS mode](https://quip-amazon.com/EUgPAQ
144146

145147
**3B: Build a System Daemon (nthd)**
146148

147-
An option to transparently handle spot interruption notifications is to build a system daemon in a separate repo that performs the IMDS monitoring and triggers an instance shutdown when an interruption is observed. This would rely on K8s’ new [graceful shutdown](https://kubernetes.io/docs/concepts/architecture/nodes/#graceful-node-shutdown) feature which went beta in K8s 1.21.
149+
An option to transparently handle spot interruption notifications is to build a system daemon in a separate repo that performs the IMDS monitoring and triggers an instance shutdown when an interruption is observed. This would rely on K8s’ new [graceful shutdown](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#graceful-node-shutdown) feature which went beta in K8s 1.21.
148150

149151
With graceful shutdown, the kubelet registers [systemd-inhibitor-locks](https://www.freedesktop.org/wiki/Software/systemd/inhibit/) to stop the shutdown flow until locks are relinquished, which in this case would be when the kubelet has drained pods off of the node. Two parameters were added to the kubelet to tune the drain timeouts: `shutdownGracePeriod` & `shutdownGracePeriodCriticalPods`
150152

‎go.mod

Lines changed: 38 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,21 @@ go 1.23.2
44

55
require (
66
github.com/Pallinder/go-randomdata v1.2.0
7-
github.com/PuerkitoBio/goquery v1.10.0
7+
github.com/PuerkitoBio/goquery v1.10.1
88
github.com/avast/retry-go v3.0.0+incompatible
9-
github.com/aws/aws-sdk-go-v2 v1.32.6
10-
github.com/aws/aws-sdk-go-v2/config v1.28.6
11-
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.21
12-
github.com/aws/aws-sdk-go-v2/service/ec2 v1.197.0
13-
github.com/aws/aws-sdk-go-v2/service/eks v1.54.0
14-
github.com/aws/aws-sdk-go-v2/service/fis v1.31.2
15-
github.com/aws/aws-sdk-go-v2/service/iam v1.38.2
16-
github.com/aws/aws-sdk-go-v2/service/pricing v1.32.7
17-
github.com/aws/aws-sdk-go-v2/service/sqs v1.37.2
18-
github.com/aws/aws-sdk-go-v2/service/ssm v1.56.1
19-
github.com/aws/aws-sdk-go-v2/service/sts v1.33.2
20-
github.com/aws/aws-sdk-go-v2/service/timestreamwrite v1.29.8
9+
github.com/aws/amazon-vpc-resource-controller-k8s v1.6.3
10+
github.com/aws/aws-sdk-go-v2 v1.32.7
11+
github.com/aws/aws-sdk-go-v2/config v1.28.7
12+
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.22
13+
github.com/aws/aws-sdk-go-v2/service/ec2 v1.198.1
14+
github.com/aws/aws-sdk-go-v2/service/eks v1.56.0
15+
github.com/aws/aws-sdk-go-v2/service/fis v1.31.3
16+
github.com/aws/aws-sdk-go-v2/service/iam v1.38.3
17+
github.com/aws/aws-sdk-go-v2/service/pricing v1.32.8
18+
github.com/aws/aws-sdk-go-v2/service/sqs v1.37.4
19+
github.com/aws/aws-sdk-go-v2/service/ssm v1.56.2
20+
github.com/aws/aws-sdk-go-v2/service/sts v1.33.3
21+
github.com/aws/aws-sdk-go-v2/service/timestreamwrite v1.29.9
2122
github.com/aws/karpenter-provider-aws/tools/kompat v0.0.0-20240410220356-6b868db24881
2223
github.com/aws/smithy-go v1.22.1
2324
github.com/awslabs/amazon-eks-ami/nodeadm v0.0.0-20240229193347-cfab22a10647
@@ -26,8 +27,8 @@ require (
2627
github.com/imdario/mergo v0.3.16
2728
github.com/jonathan-innis/aws-sdk-go-prometheus v0.1.1
2829
github.com/mitchellh/hashstructure/v2 v2.0.2
29-
github.com/onsi/ginkgo/v2 v2.22.0
30-
github.com/onsi/gomega v1.36.1
30+
github.com/onsi/ginkgo/v2 v2.22.2
31+
github.com/onsi/gomega v1.36.2
3132
github.com/patrickmn/go-cache v2.1.0+incompatible
3233
github.com/pelletier/go-toml/v2 v2.2.3
3334
github.com/prometheus/client_golang v1.20.5
@@ -42,29 +43,30 @@ require (
4243
k8s.io/client-go v0.32.0
4344
k8s.io/klog/v2 v2.130.1
4445
k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738
45-
sigs.k8s.io/controller-runtime v0.19.3
46-
sigs.k8s.io/karpenter v1.1.1
46+
sigs.k8s.io/controller-runtime v0.19.4
47+
sigs.k8s.io/karpenter v1.1.2-0.20250117235835-ff44f7325bf0
4748
sigs.k8s.io/yaml v1.4.0
4849
)
4950

5051
require (
5152
github.com/Masterminds/semver/v3 v3.2.1 // indirect
52-
github.com/andybalholm/cascadia v1.3.2 // indirect
53-
github.com/aws/aws-sdk-go-v2/credentials v1.17.47 // indirect
54-
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25 // indirect
55-
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25 // indirect
53+
github.com/andybalholm/cascadia v1.3.3 // indirect
54+
github.com/aws/aws-sdk-go-v2/credentials v1.17.48 // indirect
55+
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.26 // indirect
56+
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.26 // indirect
5657
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
5758
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
58-
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.6 // indirect
59-
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6 // indirect
60-
github.com/aws/aws-sdk-go-v2/service/sso v1.24.7 // indirect
61-
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.6 // indirect
59+
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.7 // indirect
60+
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.7 // indirect
61+
github.com/aws/aws-sdk-go-v2/service/sso v1.24.8 // indirect
62+
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.7 // indirect
6263
github.com/beorn7/perks v1.0.1 // indirect
6364
github.com/cespare/xxhash/v2 v2.3.0 // indirect
6465
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
6566
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
6667
github.com/evanphx/json-patch v5.7.0+incompatible // indirect
6768
github.com/evanphx/json-patch/v5 v5.9.0 // indirect
69+
github.com/fsnotify/fsnotify v1.7.0 // indirect
6870
github.com/fxamacker/cbor/v2 v2.7.0 // indirect
6971
github.com/go-logr/logr v1.4.2 // indirect
7072
github.com/go-openapi/jsonpointer v0.21.0 // indirect
@@ -73,10 +75,10 @@ require (
7375
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
7476
github.com/gogo/protobuf v1.3.2 // indirect
7577
github.com/golang/protobuf v1.5.4 // indirect
76-
github.com/google/gnostic-models v0.6.8 // indirect
78+
github.com/google/gnostic-models v0.6.9-0.20230804172637-c7be7c783f49 // indirect
7779
github.com/google/go-cmp v0.6.0 // indirect
7880
github.com/google/gofuzz v1.2.0 // indirect
79-
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db // indirect
81+
github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
8082
github.com/google/uuid v1.6.0 // indirect
8183
github.com/inconshreveable/mousetrap v1.1.0 // indirect
8284
github.com/jmespath/go-jmespath v0.4.0 // indirect
@@ -99,21 +101,21 @@ require (
99101
github.com/spf13/cobra v1.8.1 // indirect
100102
github.com/spf13/pflag v1.0.5 // indirect
101103
github.com/x448/float16 v0.8.4 // indirect
102-
golang.org/x/net v0.30.0 // indirect
104+
golang.org/x/net v0.33.0 // indirect
103105
golang.org/x/oauth2 v0.23.0 // indirect
104-
golang.org/x/sys v0.26.0 // indirect
105-
golang.org/x/term v0.25.0 // indirect
106-
golang.org/x/text v0.20.0 // indirect
107-
golang.org/x/time v0.8.0 // indirect
108-
golang.org/x/tools v0.26.0 // indirect
106+
golang.org/x/sys v0.28.0 // indirect
107+
golang.org/x/term v0.27.0 // indirect
108+
golang.org/x/text v0.21.0 // indirect
109+
golang.org/x/time v0.9.0 // indirect
110+
golang.org/x/tools v0.28.0 // indirect
109111
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
110-
google.golang.org/protobuf v1.35.1 // indirect
112+
google.golang.org/protobuf v1.36.1 // indirect
111113
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
112114
gopkg.in/inf.v0 v0.9.1 // indirect
113115
gopkg.in/yaml.v3 v3.0.1 // indirect
114-
k8s.io/cloud-provider v0.31.3 // indirect
116+
k8s.io/cloud-provider v0.32.0 // indirect
115117
k8s.io/component-base v0.32.0 // indirect
116-
k8s.io/csi-translation-lib v0.31.3 // indirect
118+
k8s.io/csi-translation-lib v0.32.0 // indirect
117119
k8s.io/kube-openapi v0.0.0-20241105132330-32ad38e42d3f // indirect
118120
sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect
119121
sigs.k8s.io/structured-merge-diff/v4 v4.4.2 // indirect

0 commit comments

Comments
 (0)