Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: reserved capacity updates #7804

Merged
merged 1 commit into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 68 additions & 1 deletion website/content/en/preview/concepts/nodeclasses.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,12 @@ spec:
# exclusive and can't be specified with other terms.
# - alias: al2023@v20240703

# Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
capacityReservationSelectorTerms:
- tags:
karpenter.sh/discovery: ${CLUSTER_NAME}
- id: cr-123

# Optional, propagates tags to underlying EC2 resources
tags:
team: team-a
Expand Down Expand Up @@ -188,6 +194,19 @@ status:
values:
- arm64

# Capacity Reservations
capacityReservations:
- availabilityZone: us-west-2a
id: cr-01234567890123456
instanceMatchCriteria: targeted
instanceType: g6.48xlarge
ownerID: "012345678901"
- availabilityZone: us-west-2c
id: cr-12345678901234567
instanceMatchCriteria: open
instanceType: g6.48xlarge
ownerID: "98765432109"

# Generated instance profile name from "role"
instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
conditions:
Expand Down Expand Up @@ -430,7 +449,7 @@ spec:
### AL2

{{% alert title="AL2 support dropped at Kubernetes 1.33" color="warning" %}}
Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.
Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.
{{% /alert %}}

{{% alert title="Note" color="primary" %}}
Expand Down Expand Up @@ -846,6 +865,54 @@ Specify using ids:
- id: "ami-456"
```

## spec.capacityReservationSelectorTerms

Capacity Reservation Selector Terms allow you to select [on-demand capacity reservations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html), which will be made available to NodePools which select the given EC2NodeClass.
Karpenter will prioritize utilizing the capacity in these reservations before falling back to on-demand and spot.
Capacity reservations can be discovered using ids or tags.

This selection logic is modeled as terms.
A term can specify an ID or a set of tags to select against.
When specifying tags, it will select all capacity reservations accessible from the account with matching tags.
This can be further restricted by specifying an owner ID.

#### Examples

Select the reservations with the given IDs:

```yaml
spec:
capacityReservationSelectorTerms:
- id: cr-123
- id: cr-456
```

Select the reservations by tags:

```yaml
spec:
capacityReservationSelectorTerms:
# Select all capacity reservations which have both matching tags
- tags:
key1: foo
key2: bar
# Additionally, select all capacity reservations with the following matching tag
- tags:
key3: foobar
```

Select by tags and owner ID:

```yaml
spec:
# Select all capacity reservations with the matching tags which are also owned by
# the specified account.
capacityReservationSelectorTerms:
- tags:
key: foo
ownerID: 012345678901
```

## spec.tags

Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.
Expand Down
9 changes: 5 additions & 4 deletions website/content/en/preview/concepts/nodepools.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ spec:
values: ["arm64", "amd64"]
- key: "karpenter.sh/capacity-type"
operator: In
values: ["spot", "on-demand"]
values: ["spot", "on-demand", "reserved"]

# Disruption section which describes the ways in which Karpenter can disrupt and replace Nodes
# Configuration in this section constrains how aggressive Karpenter can be with performing operations
Expand Down Expand Up @@ -251,12 +251,13 @@ Karpenter supports `linux` and `windows` operating systems.
- values
- `spot`
- `on-demand`
- `reserved`

Karpenter supports specifying capacity type, which is analogous to [EC2 purchase options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-purchasing-options.html).

Karpenter prioritizes Spot offerings if the NodePool allows Spot and on-demand instances (note that in this scenario any Spot instances priced higher than the cheapest on-demand instance will be temporarily removed from consideration).
If the provider API (e.g. EC2 Fleet's API) indicates Spot capacity is unavailable, Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes.
If there are no other possible offerings available for Spot, Karpenter will attempt to provision on-demand instances, generally within milliseconds.
If a NodePool is compatible with multiple capacity types, Karpenter will prioritize `reserved` capacity, followed by `spot`, then finally `on-demand`.
If the provider API (e.g. EC2 Fleet's API) indicates capacity is unavailable, Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes.
If there are no other possible offerings available for a higher priority capacity type, Karpenter will attempt to fallback to a lower priority capacity type, generally within milliseconds.

Karpenter also allows `karpenter.sh/capacity-type` to be used as a topology key for enforcing topology-spread.

Expand Down
2 changes: 1 addition & 1 deletion website/content/en/preview/concepts/scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ Take care to ensure the label domains are correct. A well known label like `karp
| node.kubernetes.io/windows-build | 10.0.17763 | Windows OS build in the format "MajorVersion.MinorVersion.BuildNumber". Can be `10.0.17763` for WS2019, or `10.0.20348` for WS2022. ([k8s](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesiowindows-build)) |
| kubernetes.io/os | linux | Operating systems are defined by [GOOS values](https://github.com/golang/go/blob/master/src/internal/syslist/syslist.go) (`KnownOS`) on the instance |
| kubernetes.io/arch | amd64 | Architectures are defined by [GOARCH values](https://github.com/golang/go/blob/master/src/internal/syslist/syslist.go) (`KnownArch`) on the instance |
| karpenter.sh/capacity-type | spot | Capacity types include `spot`, `on-demand` |
| karpenter.sh/capacity-type | spot | Capacity types include `reserved`, `spot`, and `on-demand` |
| karpenter.k8s.aws/instance-hypervisor | nitro | [AWS Specific] Instance types that use a specific hypervisor |
| karpenter.k8s.aws/instance-encryption-in-transit-supported | true | [AWS Specific] Instance types that support (or not) in-transit encryption |
| karpenter.k8s.aws/instance-category | g | [AWS Specific] Instance types of the same category, usually the string before the generation number |
Expand Down