Support amazon-eks-ami NodeConfig's maxPodsExpression

[sylr]

Description

What problem are you trying to solve?

I use cilium with IPAM and kube-proxy replacement and it happens that pods are scheduled on nodes that have no more IPs to give.

I tried using amazon-eks-ami NodeConfig's maxPodsExpression in my EC2NodeClass:

---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: al2023-default
spec:
  # Required, resolves a default ami and userdata
  amiFamily: AL2023
  amiSelectorTerms:
  - alias: al2023@latest

  # Required, discovers subnets to attach to instances
  subnetSelectorTerms:
  - tags:
      "kubernetes.io/cluster/eks-acme-dev-euw1-01": "owned"
      "kubernetes.io/role/internal-elb": "1"

  # Optional, overrides autogenerated userdata with a merge semantic
  userData: |
    Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
    MIME-Version: 1.0

    --MIMEBOUNDARY
    Content-Transfer-Encoding: 7bit
    Content-Type: application/node.eks.aws
    Mime-Version: 1.0

    ---
    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      cluster:
        cidr: 172.20.0.0/16
        name: eks-acme-dev-euw1-01
      kubelet:
        maxPodsExpression: "default_enis * (ips_per_eni - 1)"
        config:
          clusterDNS:
          - 172.20.0.10

    --MIMEBOUNDARY--

  # Optional, configures detailed monitoring for the instance
  detailedMonitoring: true

It calculates the wanted maxPods in /etc/kubernetes/kubelet/config.json but not in /etc/kubernetes/kubelet/config.json.d/40-nodeadm.conf

[ec2-user@ip-10-100-155-89 ~]$ cat /etc/kubernetes/kubelet/config.json | jq .maxPods
56

[ec2-user@ip-10-100-155-89 ~]$ cat /etc/kubernetes/kubelet/config.json.d/40-nodeadm.conf | jq .maxPods
58

The resulting userData for this EC2NodeClass (useless data elided):

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    cidr: 172.20.0.0/16
    name: eks-acme-staging-euc1-01
  kubelet:
    maxPodsExpression: "default_enis * (ips_per_eni - 1)"
    config:
      clusterDNS:
      - 172.20.0.10

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
metadata: {}
spec:
  cluster:
    ....
    cidr: 172.20.0.0/16
    name: eks-acme-staging-euc1-01
  containerd: {}
  instance:
    localStorage: {}
  kubelet:
    config:
      clusterDNS:
      - 172.20.0.10
      maxPods: 58
      registerWithTaints:
      - effect: NoExecute
        key: karpenter.sh/unregistered
      - effect: NoExecute
        key: node.cilium.io/agent-not-ready
        value: "true"
      - effect: NoExecute
        key: ebs.csi.aws.com/agent-not-ready
    flags:
    - --node-labels="karpenter.k8s.aws/ec2nodeclass=al2023-default,karpenter.sh/capacity-type=spot,karpenter.sh/do-not-sync-taints=true,karpenter.sh/nodepool=default-arm64"

--//--

I guess I would need the # Karpenter Generated NodeConfig to not reference maxPods in .spec.kubelet.config

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

The reason you're seeing this is because "Any MaxPods value set in Config takes precedence over the result of maxPodsExpression" (https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/api/). We're calculating the max pods in your case here with the formula max number of ENIs * (IPv4 Addresses per ENI -1) + 2. This is why the result is 58 rather than the desired 56.

I guess I would need the # Karpenter Generated NodeConfig to not reference maxPods in .spec.kubelet.config

I don't believe there is any way to currently do this.

Just to double check maxPodsExpression is not an upstream kubelet configuration right? I think the natural place for this field could be in EC2NodeClass spec.kubelet, but currently these fields are fields are a subset of the full list of upstream kubelet configuration arguments.

[sidewinder12s]

It's a nodeadm feature from the eks-ami

[ryan-mist]

I've opened a broader issue (#8742) for this as I believe theres a strong use case for expressions with kubeReserved and systemReserved. I don't believe amazon-eks-ami supports fields like that currently so we'll likely need to open up a feature request from them or explore ways to do it on Karpenters side.

The workaround currently for supporting maxPodsExpression is using a custom AMI as there is no merging of user data performed, so Karpenter won't "write in" the maxPods field. However, this does mean all the required setup must be in the UserData you provide. See https://karpenter.sh/docs/concepts/nodeclasses/#custom-2

[iciclespider]

The reason you're seeing this is because "Any MaxPods value set in Config takes precedence over the result of maxPodsExpression" (https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/api/). We're calculating the max pods in your case here with the formula max number of ENIs * (IPv4 Addresses per ENI -1) + 2. This is why the result is 58 rather than the desired 56.

@ryan-mist I just ran into this very issue myself. Can you perhaps explain to me why oh why that final + 2 is added to the default max pods calculation?

[iciclespider]

It isn't pretty, but I was able to get the correct max pods for Cilium IPAM mode set by configuring the following user data shell script in the EC2NodeClass:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  # Other fields omitted
  userData: |-
    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="//"

    --//
    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/usr/bin/env bash
    set -euo pipefail
    for config in config.json config.json.d/40-nodeadm.conf
    do
      json=$(</etc/kubernetes/kubelet/$config)
      echo "$json" | jq .maxPods=.maxPods-2 >/etc/kubernetes/kubelet/$config
    done

    --//--

[sylr]

@iciclespider I think I do even worse, I rebuild karpenter with sylr@e5e2c37

[iciclespider]

@ryan-mist I just ran into this very issue myself. Can you perhaps explain to me why oh why that final + 2 is added to the default max pods calculation?

I finally found an explanation for the + 2, it is counting both the vpc-csi and kube-proxy DaemonSet pods which run using the hostNetwork: true, and use their own ip address. Since Cilium runs with neither, this creates the situation where 2 extra ip address requiring pods can get scheduled on the nodes.

@ryan-mist Another possible solution is to provide a means to configure the number of host network DaemonSets (defaulted to 2 on AWS). I am thinking this should be a the NodePool level rather than at the NodeClass.